Aug 05 17:19:26.462249 ip-10-0-138-228 systemd[1]: Starting Kubernetes Kubelet... Aug 05 17:19:26.820820 ip-10-0-138-228 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:19:26.820820 ip-10-0-138-228 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 17:19:26.820820 ip-10-0-138-228 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:19:26.820820 ip-10-0-138-228 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 17:19:26.820820 ip-10-0-138-228 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:19:26.822380 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.822236 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 17:19:26.830499 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830477 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.830499 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830494 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.830499 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830498 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.830499 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830501 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.830499 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830505 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830510 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830514 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830517 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830520 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830523 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830526 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830530 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830534 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830536 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830539 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830542 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830544 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830547 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830549 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830552 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830554 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830557 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830560 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830562 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.830688 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830564 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830568 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830572 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830575 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830577 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830580 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830583 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830585 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830592 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830595 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830597 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830600 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830602 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830604 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830607 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830611 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830614 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830616 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830619 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830621 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.831189 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830624 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830626 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830629 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830631 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830634 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830636 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830639 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830642 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830645 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830647 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830650 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830652 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830655 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830657 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830659 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830662 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830664 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830667 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830669 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830672 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.831678 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830675 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830678 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830681 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830683 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830685 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830688 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830692 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830695 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830698 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830700 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830703 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830705 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830708 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830710 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830713 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830715 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830718 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830720 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830723 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.832177 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830726 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830735 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830738 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830740 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830742 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830745 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830747 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830749 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830752 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830754 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830757 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830759 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830762 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830764 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830766 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.830769 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831462 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831468 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831471 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.832630 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831476 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831479 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831482 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831484 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831487 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831490 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831492 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831495 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831497 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831500 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831503 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831506 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831509 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831512 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831515 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831517 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831520 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831522 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831525 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.833096 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831527 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831530 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831532 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831534 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831537 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831539 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831541 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831543 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831546 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831548 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831551 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831554 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831557 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831560 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831562 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831565 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831567 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831570 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831572 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831577 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.833565 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831580 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831583 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831586 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831589 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831591 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831594 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831597 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831599 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831602 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831605 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831607 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831609 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831612 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831614 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831616 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831619 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831621 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831624 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831626 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831628 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831631 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.834078 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831633 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831636 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831639 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831641 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831644 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831647 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831649 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831651 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831654 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831657 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831659 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831662 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831665 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831667 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831670 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831672 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831674 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831677 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831679 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.834576 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831682 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831685 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831687 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831690 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831692 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831695 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831697 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831701 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831705 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831708 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831711 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831714 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831716 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831719 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831722 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831725 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.831727 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832113 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832119 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.835033 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832122 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832125 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832128 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832131 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832134 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832136 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832139 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832142 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832144 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832147 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832149 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832152 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832154 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832157 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832160 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832163 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832165 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832167 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832170 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832172 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.835485 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832175 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832177 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832179 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832182 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832185 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832187 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832190 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832193 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832195 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832198 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832201 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832203 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832206 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832208 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832211 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832213 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832216 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832218 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832221 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832223 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.836007 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832226 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832230 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832232 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832235 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832237 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832240 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832242 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832245 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832247 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832250 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832252 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832254 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832257 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832259 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832261 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832264 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832266 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832269 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832271 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.836493 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832274 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832276 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832279 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832284 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832287 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832289 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832291 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832294 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832296 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832299 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832302 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832305 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832309 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832313 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832315 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832318 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832322 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832325 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832328 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.836949 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832331 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832334 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832337 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832340 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832343 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832346 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832348 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832351 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832353 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832356 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832358 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832360 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832363 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832365 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832368 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832370 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832374 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832376 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.832379 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832449 2564 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832456 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 17:19:26.837444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832464 2564 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832470 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832479 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832484 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832490 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832494 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832498 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832500 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832504 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832507 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832510 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832512 2564 flags.go:64] FLAG: --cgroup-root="" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832515 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832518 2564 flags.go:64] FLAG: --client-ca-file="" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832521 2564 flags.go:64] FLAG: --cloud-provider="external" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832524 2564 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832528 2564 flags.go:64] FLAG: --cluster-domain="" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832531 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832534 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832537 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832540 2564 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832543 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832546 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 17:19:26.837943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832549 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832552 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832555 2564 flags.go:64] FLAG: --contention-profiling="false" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832558 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832563 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832566 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832568 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832573 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832576 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832578 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832581 2564 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832584 2564 flags.go:64] FLAG: --enable-server="true" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832587 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832591 2564 flags.go:64] FLAG: --event-burst="100" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832594 2564 flags.go:64] FLAG: --event-qps="50" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832597 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832600 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832602 2564 flags.go:64] FLAG: --eviction-hard="" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832606 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832609 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832612 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832614 2564 flags.go:64] FLAG: --eviction-soft="" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832617 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832620 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832625 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 17:19:26.838510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832627 2564 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832630 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832633 2564 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832636 2564 flags.go:64] FLAG: --feature-gates="" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832639 2564 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832642 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832645 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832648 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832651 2564 flags.go:64] FLAG: --healthz-port="10248" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832654 2564 flags.go:64] FLAG: --help="false" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832657 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-138-228.ec2.internal" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832660 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832663 2564 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832666 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832669 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832672 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832675 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832678 2564 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832680 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832683 2564 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832686 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832688 2564 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832691 2564 flags.go:64] FLAG: --kube-reserved="" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832694 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 17:19:26.839163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832696 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832699 2564 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832701 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832704 2564 flags.go:64] FLAG: --lock-file="" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832706 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832709 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832712 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832718 2564 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832721 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832724 2564 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832727 2564 flags.go:64] FLAG: --logging-format="text" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832729 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832732 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832735 2564 flags.go:64] FLAG: --manifest-url="" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832737 2564 flags.go:64] FLAG: --manifest-url-header="" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832741 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832744 2564 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832748 2564 flags.go:64] FLAG: --max-pods="110" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832751 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832754 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832756 2564 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832759 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832762 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832765 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832768 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 17:19:26.839781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832775 2564 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832778 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832781 2564 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832784 2564 flags.go:64] FLAG: --pod-cidr="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832787 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832793 2564 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832795 2564 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832798 2564 flags.go:64] FLAG: --pods-per-core="0" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832801 2564 flags.go:64] FLAG: --port="10250" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832804 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832807 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f31f22b57b56a691" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832810 2564 flags.go:64] FLAG: --qos-reserved="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832813 2564 flags.go:64] FLAG: --read-only-port="10255" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832815 2564 flags.go:64] FLAG: --register-node="true" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832818 2564 flags.go:64] FLAG: --register-with-taints="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832822 2564 flags.go:64] FLAG: --registry-burst="10" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832826 2564 flags.go:64] FLAG: --registry-qps="5" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832829 2564 flags.go:64] FLAG: --reserved-cpus="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832831 2564 flags.go:64] FLAG: --reserved-memory="" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832834 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832837 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832840 2564 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832843 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832845 2564 flags.go:64] FLAG: --runonce="false" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832848 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 17:19:26.840406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832851 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832854 2564 flags.go:64] FLAG: --seccomp-default="false" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832856 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832859 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832862 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832865 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832867 2564 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832870 2564 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832873 2564 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832876 2564 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832878 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832881 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832884 2564 flags.go:64] FLAG: --system-cgroups="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832887 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832891 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832894 2564 flags.go:64] FLAG: --tls-cert-file="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832897 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832901 2564 flags.go:64] FLAG: --tls-min-version="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832904 2564 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832906 2564 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832909 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832912 2564 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832917 2564 flags.go:64] FLAG: --v="2" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832921 2564 flags.go:64] FLAG: --version="false" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832925 2564 flags.go:64] FLAG: --vmodule="" Aug 05 17:19:26.841042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832928 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.832932 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833061 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833064 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833068 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833071 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833074 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833076 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833080 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833085 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833087 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833091 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833093 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833096 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833099 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833102 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833105 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833108 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833111 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.841644 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833113 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833116 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833118 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833121 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833124 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833126 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833129 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833132 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833134 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833137 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833142 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833144 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833147 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833150 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833152 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833156 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833159 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833162 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833164 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833167 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.842149 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833169 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833173 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833176 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833178 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833181 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833183 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833185 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833188 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833190 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833192 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833195 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833198 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833200 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833202 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833205 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833207 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833210 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833213 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833215 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833218 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.842646 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833220 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833223 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833227 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833229 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833232 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833234 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833237 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833239 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833241 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833244 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833246 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833248 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833251 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833254 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833256 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833259 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833261 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833264 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833266 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833269 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.843183 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833271 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833274 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833276 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833279 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833281 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833283 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833286 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833288 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833290 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833293 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833295 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833298 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833300 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833302 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833306 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833308 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833311 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833313 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833317 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833322 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.843657 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833325 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.833327 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.833996 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.841096 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.841111 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.841891 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.841899 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841975 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841983 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841986 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841990 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841992 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841995 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.841998 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842001 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.844153 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842003 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842006 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842009 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842011 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842014 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842016 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842019 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842022 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842024 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842027 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842029 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842032 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842034 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842037 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842039 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842042 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842045 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842048 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842051 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842054 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.844526 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842056 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842060 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842063 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842065 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842068 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842070 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842073 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842075 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842079 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842083 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842086 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842089 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842092 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842094 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842097 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842100 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842103 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842106 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842109 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.845021 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842112 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842114 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842117 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842120 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842123 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842125 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842128 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842130 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842133 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842135 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842138 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842142 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842146 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842148 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842152 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842155 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842157 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842160 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842163 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842166 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.845572 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842168 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842171 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842173 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842176 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842178 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842181 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842183 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842186 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842188 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842191 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842193 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842195 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842198 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842200 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842203 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842205 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842207 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842210 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842212 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842215 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.846075 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842218 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842220 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842223 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842226 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842228 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842231 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842234 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842236 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842238 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842241 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842243 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842245 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.842250 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842378 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842384 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:19:26.846650 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842387 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842390 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842393 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842396 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842399 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842401 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842404 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842407 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842409 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842412 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842414 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842417 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842419 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842421 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842424 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842427 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842430 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842432 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842434 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842437 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:19:26.847185 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842439 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842442 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842445 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842447 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842450 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842453 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842455 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842457 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842460 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842462 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842464 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842467 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842469 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842471 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842474 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842476 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842479 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842481 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842483 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842486 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:19:26.847757 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842488 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842491 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842493 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842497 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842500 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842503 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842505 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842508 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842510 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842513 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842515 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842518 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842520 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842523 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842526 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842529 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842531 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842534 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842536 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842539 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:19:26.848433 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842556 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842559 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842562 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842565 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842568 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842570 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842573 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842575 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842579 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842582 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842585 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842588 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842591 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842594 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842597 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842599 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842601 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842604 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842606 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842609 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:19:26.849198 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842611 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842614 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842616 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842619 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842621 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842624 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842626 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842629 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842631 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842634 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842637 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842639 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842642 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842644 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842646 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842649 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:19:26.849679 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:26.842651 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.842656 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.842755 2564 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.845257 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.846398 2564 server.go:1019] "Starting client certificate rotation" Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.846521 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.846563 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:19:26.850127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.849105 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:19:26.850332 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.850192 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:19:26.859142 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.859122 2564 log.go:25] "Validated CRI v1 runtime API" Aug 05 17:19:26.860815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.860801 2564 log.go:25] "Validated CRI v1 image API" Aug 05 17:19:26.861540 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.861524 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 17:19:26.863715 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.863699 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f228032-3bf5-418a-80dd-916546133dd0:/dev/nvme0n1p4 d9984dbf-432b-45bb-a285-b7da73503f5c:/dev/nvme0n1p3] Aug 05 17:19:26.863769 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.863716 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 17:19:26.869092 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.868978 2564 manager.go:216] Machine: {Timestamp:2026-08-05 17:19:26.867679275 +0000 UTC m=+0.298825518 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100484 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214898d5edc69cbe01e45dee37a60e SystemUUID:ec214898-d5ed-c69c-be01-e45dee37a60e BootID:5d517fd1-b586-4a5f-b911-fd5ba58ccb95 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:76:87:3f:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:76:87:3f:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:16:fe:45:17:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 17:19:26.869092 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.869086 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 17:19:26.869199 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.869163 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 17:19:26.871649 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.871628 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 17:19:26.871786 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.871653 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 17:19:26.872353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.872341 2564 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 17:19:26.872391 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.872357 2564 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 17:19:26.872391 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.872370 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:19:26.873044 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.873035 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:19:26.873700 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.873692 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 17:19:26.876982 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.876967 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:19:26.877028 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.876985 2564 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:19:26.877121 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877111 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 17:19:26.877151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877145 2564 kubelet.go:486] "Attempting to sync node with API server" Aug 05 17:19:26.877184 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877160 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 17:19:26.877184 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877174 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 17:19:26.877184 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877182 2564 kubelet.go:398] "Adding apiserver pod source" Aug 05 17:19:26.877289 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877191 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 17:19:26.877576 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877563 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:19:26.877632 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.877585 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:19:26.880019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.880004 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 17:19:26.881251 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.881238 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 17:19:26.881291 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.881269 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 17:19:26.882100 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882087 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882107 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882115 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882120 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882129 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882134 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882142 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882147 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 17:19:26.882151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882153 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 17:19:26.882375 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882158 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 17:19:26.882375 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882177 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 17:19:26.882375 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882183 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 17:19:26.882967 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882946 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 17:19:26.883008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.882971 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 17:19:26.886120 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.886107 2564 server.go:1268] "Started kubelet" Aug 05 17:19:26.886258 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.886203 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 17:19:26.886258 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.886241 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 17:19:26.886372 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.886289 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 17:19:26.887009 ip-10-0-138-228 systemd[1]: Started Kubernetes Kubelet. Aug 05 17:19:26.888861 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.888844 2564 server.go:310] "Adding debug handlers to kubelet server" Aug 05 17:19:26.890376 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.890320 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 17:19:26.892259 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.892197 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 17:19:26.892328 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.892281 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 17:19:26.892400 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.892361 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 17:19:26.892515 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.892494 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 17:19:26.893480 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.892530 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-228.ec2.internal.18c8f86574b9305a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-228.ec2.internal,UID:ip-10-0-138-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-228.ec2.internal,},FirstTimestamp:2026-08-05 17:19:26.886080602 +0000 UTC m=+0.317226844,LastTimestamp:2026-08-05 17:19:26.886080602 +0000 UTC m=+0.317226844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-228.ec2.internal,}" Aug 05 17:19:26.894252 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894238 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 17:19:26.894304 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894254 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 17:19:26.894356 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.894340 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894747 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894807 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894766 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894945 2564 reconstruct.go:97] "Volume reconstruction finished" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.894965 2564 reconciler.go:29] "Reconciler: start to sync state" Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895171 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895185 2564 factory.go:55] Registering systemd factory Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895194 2564 factory.go:223] Registration of the systemd container factory successfully Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895346 2564 factory.go:153] Registering CRI-O factory Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895359 2564 factory.go:223] Registration of the crio container factory successfully Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895385 2564 factory.go:103] Registering Raw factory Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895399 2564 manager.go:1196] Started watching for new ooms in manager Aug 05 17:19:26.898578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.895844 2564 manager.go:349] Starting recovery of all containers Aug 05 17:19:26.899261 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.898692 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chct9" Aug 05 17:19:26.900099 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.900075 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 05 17:19:26.900220 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.900195 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 05 17:19:26.906351 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.906329 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chct9" Aug 05 17:19:26.906618 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.906599 2564 manager.go:354] Recovery completed Aug 05 17:19:26.911815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.911803 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:26.913685 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.913670 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:26.913747 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.913706 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:26.913747 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.913722 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:26.914257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.914243 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 17:19:26.914257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.914255 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 17:19:26.914359 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.914275 2564 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:19:26.914417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.914248 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:26.915118 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915105 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:26.915173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915125 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:26.915173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915134 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:26.915578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915563 2564 policy_none.go:49] "None policy: Start" Aug 05 17:19:26.915578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915580 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 17:19:26.915723 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.915591 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 17:19:26.916495 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.916482 2564 policy_none.go:47] "Start" Aug 05 17:19:26.916605 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.916466 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-228.ec2.internal.18c8f865765e6dcb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-228.ec2.internal,UID:ip-10-0-138-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-228.ec2.internal,},FirstTimestamp:2026-08-05 17:19:26.913686987 +0000 UTC m=+0.344833237,LastTimestamp:2026-08-05 17:19:26.913686987 +0000 UTC m=+0.344833237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-228.ec2.internal,}" Aug 05 17:19:26.954626 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.954458 2564 manager.go:337] "Starting Device Plugin manager" Aug 05 17:19:26.954714 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.954669 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 17:19:26.954714 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.954684 2564 server.go:87] "Starting device plugin registration server" Aug 05 17:19:26.954892 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.954882 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 17:19:26.954922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.954895 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 17:19:26.955039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.955018 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 17:19:26.955180 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.955100 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 17:19:26.955180 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:26.955109 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 17:19:26.955618 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.955597 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 17:19:26.955695 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:26.955654 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.031165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.031132 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 17:19:27.032437 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.032419 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 17:19:27.032532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.032456 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 17:19:27.032532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.032481 2564 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 17:19:27.032532 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.032522 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 17:19:27.034808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.034783 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:27.055777 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.055758 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:27.056719 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.056706 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:27.056805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.056732 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:27.056805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.056742 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:27.056805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.056764 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.062917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.062903 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.062978 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.062926 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-228.ec2.internal\": node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.078310 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.078256 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.133233 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.133205 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal"] Aug 05 17:19:27.133338 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.133265 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:27.135483 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.135468 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:27.135552 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.135501 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:27.135552 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.135514 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:27.136805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.136794 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:27.136915 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.136902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.136973 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.136935 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:27.137505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137491 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:27.137571 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137510 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:27.137571 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137520 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:27.137667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137518 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:27.137667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137620 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:27.137667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.137636 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:27.138785 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.138772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.138861 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.138795 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:19:27.140250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.140235 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:19:27.140327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.140261 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:19:27.140327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.140270 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:19:27.170648 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.170624 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-228.ec2.internal\" not found" node="ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.175002 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.174985 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-228.ec2.internal\" not found" node="ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.178422 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.178407 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.196051 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.196032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1cbf03bc1f47a41c1d8a87dcec2c129b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-228.ec2.internal\" (UID: \"1cbf03bc1f47a41c1d8a87dcec2c129b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.196132 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.196054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.196132 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.196075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.278713 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.278682 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.297089 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.297183 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.297183 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1cbf03bc1f47a41c1d8a87dcec2c129b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-228.ec2.internal\" (UID: \"1cbf03bc1f47a41c1d8a87dcec2c129b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.297183 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1cbf03bc1f47a41c1d8a87dcec2c129b-config\") pod \"kube-apiserver-proxy-ip-10-0-138-228.ec2.internal\" (UID: \"1cbf03bc1f47a41c1d8a87dcec2c129b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.297277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.297277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.297184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ed95c574ef235a7e4002e09d1838997-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal\" (UID: \"7ed95c574ef235a7e4002e09d1838997\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.379509 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.379432 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.473049 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.473014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.478459 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.478438 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.479525 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.479508 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.580038 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.580004 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.680519 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.680461 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.781041 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.781006 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.838925 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.838900 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:27.845783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.845761 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 17:19:27.845907 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.845887 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 17:19:27.845942 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.845913 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 17:19:27.881117 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:27.881092 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-228.ec2.internal\" not found" Aug 05 17:19:27.884843 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.884828 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:27.894863 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.894840 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.894936 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.894848 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 17:19:27.904492 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.904471 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:19:27.906169 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.906155 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" Aug 05 17:19:27.906831 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.906815 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:19:27.910157 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.910130 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:14:27 +0000 UTC" deadline="2028-03-09 04:08:39.130997369 +0000 UTC" Aug 05 17:19:27.910157 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.910153 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13954h49m11.22084644s" Aug 05 17:19:27.916521 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.916507 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:19:27.930624 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.930579 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-94sfs" Aug 05 17:19:27.937277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:27.937255 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-94sfs" Aug 05 17:19:28.167045 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:28.167003 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed95c574ef235a7e4002e09d1838997.slice/crio-a191463dc0c8aaac3a53c9617448c3c3c83964b5718e0c6333d1d8a19e1fb96a WatchSource:0}: Error finding container a191463dc0c8aaac3a53c9617448c3c3c83964b5718e0c6333d1d8a19e1fb96a: Status 404 returned error can't find the container with id a191463dc0c8aaac3a53c9617448c3c3c83964b5718e0c6333d1d8a19e1fb96a Aug 05 17:19:28.167307 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:28.167288 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbf03bc1f47a41c1d8a87dcec2c129b.slice/crio-f242dd500e975a6350d4af6a5919f54df24cac4610aa224470b953886566cbce WatchSource:0}: Error finding container f242dd500e975a6350d4af6a5919f54df24cac4610aa224470b953886566cbce: Status 404 returned error can't find the container with id f242dd500e975a6350d4af6a5919f54df24cac4610aa224470b953886566cbce Aug 05 17:19:28.171825 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.171804 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:19:28.289394 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.289320 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:28.878145 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.878110 2564 apiserver.go:52] "Watching apiserver" Aug 05 17:19:28.887994 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.887967 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 17:19:28.888542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.888498 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-n7zx2","openshift-image-registry/node-ca-q8rkq","openshift-multus/multus-additional-cni-plugins-8gsgs","openshift-multus/multus-pklxj","openshift-ovn-kubernetes/ovnkube-node-h8r2f","kube-system/global-pull-secret-syncer-vb66t","kube-system/konnectivity-agent-2jfxr","openshift-dns/node-resolver-q9rcr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal","openshift-multus/network-metrics-daemon-2hc9s","openshift-network-diagnostics/network-check-target-5x96h","openshift-network-operator/iptables-alerter-d4lcs","kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp"] Aug 05 17:19:28.890352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.890324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:28.892665 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.892641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.893315 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.893292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 17:19:28.893420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.893395 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5dt47\"" Aug 05 17:19:28.893677 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.893658 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 17:19:28.896059 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.894930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:28.896059 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.895384 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.896059 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.895696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d9zh9\"" Aug 05 17:19:28.896059 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.895903 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.896535 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.896475 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.897578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.897555 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.897949 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.897676 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.897949 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.897879 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvpwb\"" Aug 05 17:19:28.898478 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.898458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:28.898870 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.898849 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 17:19:28.899386 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.899367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.899491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.899398 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.899491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.899450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 17:19:28.899602 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.899591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s5slm\"" Aug 05 17:19:28.900637 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.900618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:28.900732 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:28.900685 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:28.900869 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.900851 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.900961 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.900946 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 17:19:28.901152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.901118 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 17:19:28.902361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.902064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:28.903143 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.903006 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 17:19:28.903903 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.903283 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xpz8w\"" Aug 05 17:19:28.903903 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.903704 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.903903 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.903705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 17:19:28.904584 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.904320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:28.904584 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:28.904394 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:28.904719 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.904672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzc6c\"" Aug 05 17:19:28.904765 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.904714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.904800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.904776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 17:19:28.905740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.905602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:28.905740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.905644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.905740 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:28.905661 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:28.906722 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-modprobe-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.906805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysconfig\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.906805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-tmp-dir\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:28.906805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-cni-binary-copy\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.906940 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-kubelet\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.906940 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmfj\" (UniqueName: \"kubernetes.io/projected/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-kube-api-access-5fmfj\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:28.906940 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.906850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-cnibin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.907214 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-agent-certs\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:28.907295 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-kubernetes\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-conf\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-tuned\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhh5\" (UniqueName: \"kubernetes.io/projected/bf948251-0270-46a3-9aaa-e4dcfffa4890-kube-api-access-hmhh5\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-multus-daemon-config\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.907455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-systemd\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-socket-dir-parent\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.907612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:28.907612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97wv\" (UniqueName: \"kubernetes.io/projected/d793d02b-2909-4792-a697-cf7b93a36426-kube-api-access-p97wv\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.907612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-lib-modules\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.907612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-konnectivity-ca\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.907600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-host\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-tmp\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-system-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-netns\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-conf-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-var-lib-kubelet\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-hosts-file\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-os-release\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-run\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-k8s-cni-cncf-io\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.908997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-bin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909699 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-multus\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909699 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-hostroot\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909853 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-multus-certs\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909853 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-etc-kubernetes\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.909853 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-sys\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:28.910001 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.909857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:28.911840 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.910854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.911840 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.910873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.911840 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.911178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:28.911840 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.911207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9vsw\"" Aug 05 17:19:28.912117 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.911883 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 17:19:28.913551 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.913529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:28.914010 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.913870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-584xh\"" Aug 05 17:19:28.914701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.914683 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 17:19:28.914800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.914778 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 17:19:28.916244 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.916212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 17:19:28.916912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.916710 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 17:19:28.916912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.916727 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 17:19:28.916912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.916787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x2jbd\"" Aug 05 17:19:28.939111 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.938989 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:14:28 +0000 UTC" deadline="2028-01-01 09:44:03.353839813 +0000 UTC" Aug 05 17:19:28.939111 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.939110 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12328h24m34.414734363s" Aug 05 17:19:28.996654 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:28.996614 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 17:19:29.010050 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-os-release\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-serviceca\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-device-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-kubelet\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-etc-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.010204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-os-release\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-node-log\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-run\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b3a8fd2-3454-4fd3-9633-daefe1401e40-host-slash\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-run\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-registration-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.010488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-env-overrides\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-dbus\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-bin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-hostroot\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-bin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-etc-kubernetes\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-etc-kubernetes\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cg7q\" (UniqueName: \"kubernetes.io/projected/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-kube-api-access-2cg7q\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-sys\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-hostroot\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-sys\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.010827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cnibin\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdd4\" (UniqueName: \"kubernetes.io/projected/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-kube-api-access-2rdd4\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-netd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-config\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-modprobe-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.010973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysconfig\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-tmp-dir\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-kubelet\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5pl\" (UniqueName: \"kubernetes.io/projected/ce21a759-3afa-4624-a512-c3b5b52cb014-kube-api-access-8g5pl\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-modprobe-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-ovn\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-kubelet\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-kubernetes\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-conf\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysconfig\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvhr\" (UniqueName: \"kubernetes.io/projected/9b3a8fd2-3454-4fd3-9633-daefe1401e40-kube-api-access-mmvhr\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-kubernetes\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmd8\" (UniqueName: \"kubernetes.io/projected/a3311739-a779-4c3d-a848-5482e3cee472-kube-api-access-cvmd8\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-slash\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-conf\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmx7\" (UniqueName: \"kubernetes.io/projected/699e58a9-31c5-4a21-a7e9-207f2108e7f8-kube-api-access-wsmx7\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97wv\" (UniqueName: \"kubernetes.io/projected/d793d02b-2909-4792-a697-cf7b93a36426-kube-api-access-p97wv\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-tmp-dir\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-netns\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-lib-modules\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-host\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-systemd-units\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-var-lib-kubelet\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.011978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-systemd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-host\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b3a8fd2-3454-4fd3-9633-daefe1401e40-iptables-alerter-script\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-lib-modules\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-script-lib\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-k8s-cni-cncf-io\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-var-lib-kubelet\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-sysctl-d\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-multus\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-k8s-cni-cncf-io\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-var-lib-cni-multus\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-multus-certs\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-os-release\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.011976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-multus-certs\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-bin\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.012749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-sys-fs\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-log-socket\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-cni-binary-copy\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovn-node-metrics-cert\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmfj\" (UniqueName: \"kubernetes.io/projected/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-kube-api-access-5fmfj\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-cnibin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-agent-certs\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-tuned\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhh5\" (UniqueName: \"kubernetes.io/projected/bf948251-0270-46a3-9aaa-e4dcfffa4890-kube-api-access-hmhh5\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-multus-daemon-config\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-cnibin\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-systemd\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.013616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-socket-dir-parent\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-host\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-var-lib-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-konnectivity-ca\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-tmp\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-system-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-netns\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-cni-binary-copy\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012911 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-conf-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-socket-dir-parent\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.012985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-multus-conf-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-socket-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-kubelet-config\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-hosts-file\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-hosts-file\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-systemd\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-host-run-netns\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.014276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d793d02b-2909-4792-a697-cf7b93a36426-system-cni-dir\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.015173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d793d02b-2909-4792-a697-cf7b93a36426-multus-daemon-config\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.015173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.013586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-konnectivity-ca\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:29.017283 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.017249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-etc-tuned\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.017392 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.017285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf948251-0270-46a3-9aaa-e4dcfffa4890-tmp\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.017852 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.017831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b936f9c2-6a92-4cfa-8130-8c9573c4b58f-agent-certs\") pod \"konnectivity-agent-2jfxr\" (UID: \"b936f9c2-6a92-4cfa-8130-8c9573c4b58f\") " pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:29.023787 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.023715 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97wv\" (UniqueName: \"kubernetes.io/projected/d793d02b-2909-4792-a697-cf7b93a36426-kube-api-access-p97wv\") pod \"multus-pklxj\" (UID: \"d793d02b-2909-4792-a697-cf7b93a36426\") " pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.024158 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.024130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhh5\" (UniqueName: \"kubernetes.io/projected/bf948251-0270-46a3-9aaa-e4dcfffa4890-kube-api-access-hmhh5\") pod \"tuned-n7zx2\" (UID: \"bf948251-0270-46a3-9aaa-e4dcfffa4890\") " pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.025182 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.025154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmfj\" (UniqueName: \"kubernetes.io/projected/e6d3d3d4-1f92-48ad-8aeb-7bea6f342378-kube-api-access-5fmfj\") pod \"node-resolver-q9rcr\" (UID: \"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378\") " pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.038358 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.038310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" event={"ID":"7ed95c574ef235a7e4002e09d1838997","Type":"ContainerStarted","Data":"a191463dc0c8aaac3a53c9617448c3c3c83964b5718e0c6333d1d8a19e1fb96a"} Aug 05 17:19:29.039784 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.039754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" event={"ID":"1cbf03bc1f47a41c1d8a87dcec2c129b","Type":"ContainerStarted","Data":"f242dd500e975a6350d4af6a5919f54df24cac4610aa224470b953886566cbce"} Aug 05 17:19:29.058902 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.058876 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:29.114434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-systemd-units\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.114434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-systemd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b3a8fd2-3454-4fd3-9633-daefe1401e40-iptables-alerter-script\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-systemd-units\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114503 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-systemd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-script-lib\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-os-release\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-bin\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-sys-fs\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-os-release\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-log-socket\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.114737 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:29.114760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.114840 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:29.614794358 +0000 UTC m=+3.045940608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovn-node-metrics-cert\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.114991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-host\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-var-lib-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-socket-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-kubelet-config\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-serviceca\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-device-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-kubelet\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-etc-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.115421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b3a8fd2-3454-4fd3-9633-daefe1401e40-iptables-alerter-script\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-node-log\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-log-socket\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b3a8fd2-3454-4fd3-9633-daefe1401e40-host-slash\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b3a8fd2-3454-4fd3-9633-daefe1401e40-host-slash\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-sys-fs\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-registration-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-env-overrides\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-dbus\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-socket-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cg7q\" (UniqueName: \"kubernetes.io/projected/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-kube-api-access-2cg7q\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-dbus\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cnibin\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.116307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdd4\" (UniqueName: \"kubernetes.io/projected/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-kube-api-access-2rdd4\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8246d4ad-44f1-4c46-8f31-62348f5a71d3-kubelet-config\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-netd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-bin\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-device-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-script-lib\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-config\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5pl\" (UniqueName: \"kubernetes.io/projected/ce21a759-3afa-4624-a512-c3b5b52cb014-kube-api-access-8g5pl\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-ovn\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.115959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-registration-dir\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-host\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce21a759-3afa-4624-a512-c3b5b52cb014-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-var-lib-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-system-cni-dir\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.117136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-kubelet\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-etc-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-serviceca\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-cni-netd\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-node-log\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvhr\" (UniqueName: \"kubernetes.io/projected/9b3a8fd2-3454-4fd3-9633-daefe1401e40-kube-api-access-mmvhr\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-ovn\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmd8\" (UniqueName: \"kubernetes.io/projected/a3311739-a779-4c3d-a848-5482e3cee472-kube-api-access-cvmd8\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cnibin\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovnkube-config\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/699e58a9-31c5-4a21-a7e9-207f2108e7f8-env-overrides\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-slash\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-slash\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmx7\" (UniqueName: \"kubernetes.io/projected/699e58a9-31c5-4a21-a7e9-207f2108e7f8-kube-api-access-wsmx7\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-run-openvswitch\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.117712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-netns\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.118378 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.116939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.118378 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.117018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/699e58a9-31c5-4a21-a7e9-207f2108e7f8-host-run-netns\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.118378 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.117061 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:29.118378 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.117128 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:29.617108561 +0000 UTC m=+3.048254794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:29.118378 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.117060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-cni-binary-copy\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.119930 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.119904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/699e58a9-31c5-4a21-a7e9-207f2108e7f8-ovn-node-metrics-cert\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.124389 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.124018 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:29.124389 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.124049 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:29.124389 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.124063 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:29.124389 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.124123 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:29.624105812 +0000 UTC m=+3.055252059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:29.127010 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.126984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvhr\" (UniqueName: \"kubernetes.io/projected/9b3a8fd2-3454-4fd3-9633-daefe1401e40-kube-api-access-mmvhr\") pod \"iptables-alerter-d4lcs\" (UID: \"9b3a8fd2-3454-4fd3-9633-daefe1401e40\") " pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.127889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.127863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cg7q\" (UniqueName: \"kubernetes.io/projected/4e67ba62-36b0-4223-83f8-00eb8fbc53b9-kube-api-access-2cg7q\") pod \"node-ca-q8rkq\" (UID: \"4e67ba62-36b0-4223-83f8-00eb8fbc53b9\") " pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.130078 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.129979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5pl\" (UniqueName: \"kubernetes.io/projected/ce21a759-3afa-4624-a512-c3b5b52cb014-kube-api-access-8g5pl\") pod \"aws-ebs-csi-driver-node-lf2vp\" (UID: \"ce21a759-3afa-4624-a512-c3b5b52cb014\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.130285 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.130263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmd8\" (UniqueName: \"kubernetes.io/projected/a3311739-a779-4c3d-a848-5482e3cee472-kube-api-access-cvmd8\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.130423 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.130401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdd4\" (UniqueName: \"kubernetes.io/projected/aaf06d87-8c29-4f6d-8c51-0d9834cab5ea-kube-api-access-2rdd4\") pod \"multus-additional-cni-plugins-8gsgs\" (UID: \"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea\") " pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.133355 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.133327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmx7\" (UniqueName: \"kubernetes.io/projected/699e58a9-31c5-4a21-a7e9-207f2108e7f8-kube-api-access-wsmx7\") pod \"ovnkube-node-h8r2f\" (UID: \"699e58a9-31c5-4a21-a7e9-207f2108e7f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.209530 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.209495 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:29.218424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.218403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pklxj" Aug 05 17:19:29.227257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.227234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" Aug 05 17:19:29.233875 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.233847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9rcr" Aug 05 17:19:29.242713 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.242694 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:29.249651 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.249629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8rkq" Aug 05 17:19:29.259405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.259387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4lcs" Aug 05 17:19:29.268202 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.268180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" Aug 05 17:19:29.272846 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.272819 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" Aug 05 17:19:29.621577 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.621479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:29.621727 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.621595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:29.621727 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.621628 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:29.621727 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.621693 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:30.621677678 +0000 UTC m=+4.052823907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:29.621727 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.621704 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:29.621897 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.621757 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:30.621739063 +0000 UTC m=+4.052885298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:29.722442 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.722409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:29.722596 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.722579 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:29.722637 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.722599 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:29.722637 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.722610 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:29.722708 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:29.722663 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:30.722647455 +0000 UTC m=+4.153793686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:29.939836 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.939723 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:14:28 +0000 UTC" deadline="2028-05-06 22:34:28.074241384 +0000 UTC" Aug 05 17:19:29.939836 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:29.939773 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15365h14m58.13447164s" Aug 05 17:19:30.033580 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:30.033550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:30.033749 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.033676 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:30.166352 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:30.166274 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3a8fd2_3454_4fd3_9633_daefe1401e40.slice/crio-b389a9d3ff8fc42965f93602d3b4274eda9936f0f46b86e233580b58fefe3f59 WatchSource:0}: Error finding container b389a9d3ff8fc42965f93602d3b4274eda9936f0f46b86e233580b58fefe3f59: Status 404 returned error can't find the container with id b389a9d3ff8fc42965f93602d3b4274eda9936f0f46b86e233580b58fefe3f59 Aug 05 17:19:30.167541 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:19:30.167515 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce21a759_3afa_4624_a512_c3b5b52cb014.slice/crio-ac8588caac87a10677a1f3b13bac1f408452308c0b3f1836e0062e34fbd2f2d3 WatchSource:0}: Error finding container ac8588caac87a10677a1f3b13bac1f408452308c0b3f1836e0062e34fbd2f2d3: Status 404 returned error can't find the container with id ac8588caac87a10677a1f3b13bac1f408452308c0b3f1836e0062e34fbd2f2d3 Aug 05 17:19:30.630503 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:30.630292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:30.630694 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:30.630533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:30.630694 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.630653 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:30.630820 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.630715 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:32.630696174 +0000 UTC m=+6.061842404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:30.631289 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.631168 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:30.631289 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.631227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:32.631210258 +0000 UTC m=+6.062356492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:30.731000 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:30.730910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:30.731182 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.731122 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:30.731182 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.731146 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:30.731182 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.731159 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:30.731342 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:30.731222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:32.731202569 +0000 UTC m=+6.162348821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:30.998588 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:30.998508 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:19:31.035803 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.035768 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:31.035989 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:31.035926 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:31.036363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.036340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:31.036463 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:31.036445 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:31.064948 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.064876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4lcs" event={"ID":"9b3a8fd2-3454-4fd3-9633-daefe1401e40","Type":"ContainerStarted","Data":"b389a9d3ff8fc42965f93602d3b4274eda9936f0f46b86e233580b58fefe3f59"} Aug 05 17:19:31.072777 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.072740 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ed95c574ef235a7e4002e09d1838997" containerID="96508cff4b948f08c660ce79d71c176d56e92ec8539f3d0b1ba71ab4585796a3" exitCode=0 Aug 05 17:19:31.073453 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.073413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" event={"ID":"7ed95c574ef235a7e4002e09d1838997","Type":"ContainerDied","Data":"96508cff4b948f08c660ce79d71c176d56e92ec8539f3d0b1ba71ab4585796a3"} Aug 05 17:19:31.087261 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.087166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"6564e61ccd299de77cd196e6b28aea12596661b85ff1b975941f746500f621a0"} Aug 05 17:19:31.094926 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.093147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pklxj" event={"ID":"d793d02b-2909-4792-a697-cf7b93a36426","Type":"ContainerStarted","Data":"6a80bd49ebd00df664a86bec858ba7a5a1a1cc277787adf616328d803c9b7dcc"} Aug 05 17:19:31.101208 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.101090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2jfxr" event={"ID":"b936f9c2-6a92-4cfa-8130-8c9573c4b58f","Type":"ContainerStarted","Data":"cf9ea7b3155488060884637e5016103617ef2d5a0eb16b7b530875473aeef72c"} Aug 05 17:19:31.116103 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.116073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerStarted","Data":"a19d5cb3bc412bc486b6961abffcf754a0fe2ae345d40adad9aa427bc91afdf7"} Aug 05 17:19:31.128557 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.128520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" event={"ID":"ce21a759-3afa-4624-a512-c3b5b52cb014","Type":"ContainerStarted","Data":"ac8588caac87a10677a1f3b13bac1f408452308c0b3f1836e0062e34fbd2f2d3"} Aug 05 17:19:31.136481 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.136451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" event={"ID":"1cbf03bc1f47a41c1d8a87dcec2c129b","Type":"ContainerStarted","Data":"58feb05c865af79f8b57e5e1ab9d18a927aff375a51a81b6f20beeaaa5a65876"} Aug 05 17:19:31.150583 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.150554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" event={"ID":"bf948251-0270-46a3-9aaa-e4dcfffa4890","Type":"ContainerStarted","Data":"f907d83b6e0d7fac0b45a7b0a0d8b77776382a474cf79e8133f9aa156b274cae"} Aug 05 17:19:31.156257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.155062 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-228.ec2.internal" podStartSLOduration=3.155027387 podStartE2EDuration="3.155027387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:31.154861793 +0000 UTC m=+4.586008051" watchObservedRunningTime="2026-08-05 17:19:31.155027387 +0000 UTC m=+4.586173642" Aug 05 17:19:31.159129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.159106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9rcr" event={"ID":"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378","Type":"ContainerStarted","Data":"4e52e931e7d22f683ad9db661f1a78e09bdd0cff0276b47681bf28ae993826ee"} Aug 05 17:19:31.165884 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:31.165856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8rkq" event={"ID":"4e67ba62-36b0-4223-83f8-00eb8fbc53b9","Type":"ContainerStarted","Data":"c87e009e80b175d689bef471ac6b4b537abd2e7ae786721652cf092f44c8f454"} Aug 05 17:19:32.033321 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:32.033285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:32.033807 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.033436 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:32.196199 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:32.195862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" event={"ID":"7ed95c574ef235a7e4002e09d1838997","Type":"ContainerStarted","Data":"11682bc2623c264077335c9f7a10f6289fa8a46b11492c92edba27f71906fd7c"} Aug 05 17:19:32.653241 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:32.653075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:32.653241 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:32.653147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:32.653241 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.653215 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:32.653545 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.653294 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:36.653269223 +0000 UTC m=+10.084415463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:32.653545 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.653294 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:32.653545 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.653336 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:36.65332473 +0000 UTC m=+10.084470960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:32.754522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:32.753873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:32.754522 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.754058 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:32.754522 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.754077 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:32.754522 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.754090 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:32.754522 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:32.754151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:36.754130992 +0000 UTC m=+10.185277242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:33.036178 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:33.035756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:33.036178 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:33.035900 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:33.036178 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:33.036160 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:33.036789 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:33.036287 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:34.032824 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:34.032679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:34.032824 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:34.032809 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:35.035641 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:35.035611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:35.036146 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:35.035746 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:35.036146 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:35.036142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:35.036271 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:35.036245 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:36.033241 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:36.033205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:36.033442 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.033382 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:36.689162 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:36.689108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:36.689162 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:36.689167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:36.689721 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.689288 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:36.689721 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.689352 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:44.689332255 +0000 UTC m=+18.120478488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:36.689821 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.689723 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:36.689821 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.689764 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:44.689752225 +0000 UTC m=+18.120898454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:36.790496 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:36.790444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:36.790644 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.790617 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:36.790644 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.790637 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:36.790724 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.790649 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:36.790724 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:36.790707 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:44.790689 +0000 UTC m=+18.221835248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:37.034181 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:37.034098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:37.034346 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:37.034229 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:37.034683 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:37.034664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:37.034772 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:37.034748 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:38.034278 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:38.033764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:38.034278 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:38.033913 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:39.035917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:39.035159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:39.035917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:39.035158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:39.035917 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:39.035293 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:39.035917 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:39.035397 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:40.033631 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:40.033596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:40.033829 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:40.033745 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:41.032789 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:41.032750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:41.033249 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:41.032884 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:41.033249 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:41.032939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:41.033249 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:41.033069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:42.033675 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:42.033639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:42.034148 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:42.033752 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:43.033045 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:43.033014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:43.033279 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:43.033051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:43.033279 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:43.033157 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:43.033385 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:43.033284 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:44.033144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:44.033117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:44.033600 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.033227 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:44.758987 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:44.758929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:44.759196 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:44.759016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:44.759196 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.759080 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:44.759196 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.759135 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:44.759196 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.759151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:00.759132247 +0000 UTC m=+34.190278487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:19:44.759196 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.759189 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:00.759172066 +0000 UTC m=+34.190318317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:19:44.859572 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:44.859542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:44.859736 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.859676 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:19:44.859736 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.859694 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:19:44.859736 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.859707 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:44.859886 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:44.859766 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:00.859748391 +0000 UTC m=+34.290894624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:19:45.033129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:45.033047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:45.033293 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:45.033053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:45.033293 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:45.033176 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:45.033293 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:45.033261 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:46.032994 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:46.032940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:46.033158 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:46.033070 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:47.034636 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:47.034602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:47.035122 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:47.034758 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:47.035122 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:47.034792 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:47.035122 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:47.034889 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:48.032775 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:48.032735 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:48.032937 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:48.032852 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:49.033060 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:49.033029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:49.033474 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:49.033156 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:49.033474 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:49.033243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:49.033474 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:49.033345 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:50.033550 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.033248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:50.034293 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:50.033647 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:50.237623 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.237590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" event={"ID":"ce21a759-3afa-4624-a512-c3b5b52cb014","Type":"ContainerStarted","Data":"5968ab6c3236c8cfd3d369f87440cd21247c637b9a5b9f0f62905273920a1da4"} Aug 05 17:19:50.239016 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.238986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" event={"ID":"bf948251-0270-46a3-9aaa-e4dcfffa4890","Type":"ContainerStarted","Data":"346558f99170a9d3b21871434352b0f6aeaedc466bf4e30c86eb313a59f3625e"} Aug 05 17:19:50.240372 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.240346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9rcr" event={"ID":"e6d3d3d4-1f92-48ad-8aeb-7bea6f342378","Type":"ContainerStarted","Data":"5df2a2f674e11668e2a8b02bc1829e246952f02401fbe3739a45264c0706fef5"} Aug 05 17:19:50.241677 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.241653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8rkq" event={"ID":"4e67ba62-36b0-4223-83f8-00eb8fbc53b9","Type":"ContainerStarted","Data":"a6d55396e67aff852b245d7087e61d01335588361c40b480c792e586ae47782b"} Aug 05 17:19:50.244466 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"98bf841ef48feeac0137f15ec3364ee14b57a614f13c1ec289c9626addeb690d"} Aug 05 17:19:50.244561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"b47c3a2f3c652662316fa33bb73ad5f5b5a5b5a730dff56e7d1e961f8f0960c5"} Aug 05 17:19:50.244561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"2acce1e45c77488cc53f33dd2f831c7736e6d497536916b3e48238149d46e466"} Aug 05 17:19:50.244672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"6dac67b19962598cd6afa188619f15733000a31b49915e4d51fe03cac7700690"} Aug 05 17:19:50.244672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"77b0a14cf49a3464fcd10ff2c7009d6097695ec6121231f89c10fd359b2d49a6"} Aug 05 17:19:50.244672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.244644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"9c6a16594657652ce2d5311a09c28d785d09783d5ad526d61f7444c9bc33d08a"} Aug 05 17:19:50.245907 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.245882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pklxj" event={"ID":"d793d02b-2909-4792-a697-cf7b93a36426","Type":"ContainerStarted","Data":"00fd62335c4faede5ca209d82fa237d79887a35640a800d765e4941ea3f17a52"} Aug 05 17:19:50.247227 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.247178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2jfxr" event={"ID":"b936f9c2-6a92-4cfa-8130-8c9573c4b58f","Type":"ContainerStarted","Data":"46ed522b0e1a2465c5b89ad4f4ea183091ab2534124c92f03008b2380d39c5b0"} Aug 05 17:19:50.248550 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.248530 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="9abcff4f753521156877ada5d5fe867580fbb9b03052a4ab8aaf632484fc3b19" exitCode=0 Aug 05 17:19:50.248640 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.248556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"9abcff4f753521156877ada5d5fe867580fbb9b03052a4ab8aaf632484fc3b19"} Aug 05 17:19:50.276383 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.276340 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-228.ec2.internal" podStartSLOduration=22.276327333 podStartE2EDuration="22.276327333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:32.223328596 +0000 UTC m=+5.654474852" watchObservedRunningTime="2026-08-05 17:19:50.276327333 +0000 UTC m=+23.707473582" Aug 05 17:19:50.306128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.306089 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n7zx2" podStartSLOduration=4.257602391 podStartE2EDuration="23.306075897s" totalImagesPullingTime="19.048473506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:50.276151841 +0000 UTC m=+23.707298090" watchObservedRunningTime="2026-08-05 17:19:50.306075897 +0000 UTC m=+23.737222172" Aug 05 17:19:50.306444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.306423 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2jfxr" podStartSLOduration=8.176987986 podStartE2EDuration="23.306418571s" totalImagesPullingTime="15.129430585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:50.305871467 +0000 UTC m=+23.737017716" watchObservedRunningTime="2026-08-05 17:19:50.306418571 +0000 UTC m=+23.737564821" Aug 05 17:19:50.321467 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.321418 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q8rkq" podStartSLOduration=4.284437505 podStartE2EDuration="23.321394733s" totalImagesPullingTime="19.036957228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:50.321277193 +0000 UTC m=+23.752423445" watchObservedRunningTime="2026-08-05 17:19:50.321394733 +0000 UTC m=+23.752540986" Aug 05 17:19:50.384797 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.384758 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q9rcr" podStartSLOduration=4.353177799 podStartE2EDuration="23.384745538s" totalImagesPullingTime="19.031567739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:50.364675245 +0000 UTC m=+23.795821506" watchObservedRunningTime="2026-08-05 17:19:50.384745538 +0000 UTC m=+23.815891790" Aug 05 17:19:50.385145 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.385122 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pklxj" podStartSLOduration=4.277978119 podStartE2EDuration="23.385114367s" totalImagesPullingTime="19.107136248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:50.384599297 +0000 UTC m=+23.815745582" watchObservedRunningTime="2026-08-05 17:19:50.385114367 +0000 UTC m=+23.816260616" Aug 05 17:19:50.502491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.502466 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 17:19:50.911036 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.910993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:50.911607 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.911586 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:50.967703 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.966852 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:19:50.502487629Z","UUID":"f4c2684e-6b6f-441c-8d83-b5de94b4d050","Handler":null,"Name":"","Endpoint":""} Aug 05 17:19:50.968829 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.968797 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 17:19:50.968972 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:50.968842 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 17:19:51.032917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.032887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:51.033094 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:51.033018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:51.033341 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.033325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:51.033453 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:51.033435 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:51.253784 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.253704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" event={"ID":"ce21a759-3afa-4624-a512-c3b5b52cb014","Type":"ContainerStarted","Data":"8041bb24ee3534f723cc86425d9152428b0c7f5f491216bb26cccf24f1ad01ce"} Aug 05 17:19:51.255517 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.255490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4lcs" event={"ID":"9b3a8fd2-3454-4fd3-9633-daefe1401e40","Type":"ContainerStarted","Data":"0be1fe65560576b5709a7eb920281d6ef8e2c85bede588cc6ccfeb333be22e91"} Aug 05 17:19:51.256873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.256849 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:51.257364 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.257345 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2jfxr" Aug 05 17:19:51.289579 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:51.289536 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d4lcs" podStartSLOduration=4.238863995 podStartE2EDuration="23.289519254s" totalImagesPullingTime="19.050655259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:51.272046999 +0000 UTC m=+24.703193248" watchObservedRunningTime="2026-08-05 17:19:51.289519254 +0000 UTC m=+24.720665506" Aug 05 17:19:52.033397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:52.033322 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:52.033560 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:52.033433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:52.260210 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:52.260182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" event={"ID":"ce21a759-3afa-4624-a512-c3b5b52cb014","Type":"ContainerStarted","Data":"48f419991b5282c21b2510158c0667eb0b36d40b65b11b3f521743d6bd6a3d85"} Aug 05 17:19:53.033246 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:53.033211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:53.033428 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:53.033347 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:53.033428 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:53.033402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:53.033551 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:53.033529 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:53.265894 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:53.265858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"a9c70aec20b467021e49f9a7b366396ef1e2fb733a00baae59563dc9cdeed547"} Aug 05 17:19:54.033605 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:54.033577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:54.033780 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:54.033696 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:55.033009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.032834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:55.033580 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.032891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:55.033580 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:55.033095 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:55.033580 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:55.033170 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:55.273229 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.273190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" event={"ID":"699e58a9-31c5-4a21-a7e9-207f2108e7f8","Type":"ContainerStarted","Data":"b71121cf5991d901f473a29d6cc234b8c9733e2286b8b8ebff2167ad20ee0fb7"} Aug 05 17:19:55.273542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.273524 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:55.273690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.273578 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:55.274935 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.274906 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="725da5a2f659a1dc54d6542eb8cc11c1fc4917acd1aebb0cec7b7225857ba1ce" exitCode=0 Aug 05 17:19:55.275036 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.274945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"725da5a2f659a1dc54d6542eb8cc11c1fc4917acd1aebb0cec7b7225857ba1ce"} Aug 05 17:19:55.289327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.289310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:55.302809 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.302774 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" podStartSLOduration=9.212251552 podStartE2EDuration="28.30276085s" totalImagesPullingTime="19.090509298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:55.302729052 +0000 UTC m=+28.733875301" watchObservedRunningTime="2026-08-05 17:19:55.30276085 +0000 UTC m=+28.733907090" Aug 05 17:19:55.303182 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:55.303161 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf2vp" podStartSLOduration=6.238185836 podStartE2EDuration="27.30315422s" totalImagesPullingTime="21.064968384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:52.285285615 +0000 UTC m=+25.716431865" watchObservedRunningTime="2026-08-05 17:19:55.30315422 +0000 UTC m=+28.734300470" Aug 05 17:19:56.033842 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.033653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:56.034388 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:56.033926 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:56.277088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.277062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:56.293130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.293095 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:19:56.605142 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.605108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb66t"] Aug 05 17:19:56.605312 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.605230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:56.605372 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:56.605338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:56.609048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.609026 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5x96h"] Aug 05 17:19:56.609187 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.609117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:56.609242 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:56.609193 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:56.609833 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.609813 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hc9s"] Aug 05 17:19:56.609922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:56.609900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:56.610030 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:56.610009 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:57.280752 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:57.280720 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="95a30d7ca1ed0183781344115f993091ff10455af70f6de1f8aa05c5ae59f371" exitCode=0 Aug 05 17:19:57.281251 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:57.280804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"95a30d7ca1ed0183781344115f993091ff10455af70f6de1f8aa05c5ae59f371"} Aug 05 17:19:58.033144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:58.033118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:19:58.033304 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:58.033223 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:19:58.692073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:58.692006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:58.800048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:58.799835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:59.035652 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:59.035578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:19:59.035789 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:59.035709 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:19:59.035789 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:59.035753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:19:59.035861 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:19:59.035823 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:19:59.287440 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:59.287360 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="abd093204ac48afc984a87b5b93ae140a95f4c3317a40817e414f690ec99ee08" exitCode=0 Aug 05 17:19:59.287440 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:19:59.287404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"abd093204ac48afc984a87b5b93ae140a95f4c3317a40817e414f690ec99ee08"} Aug 05 17:20:00.033433 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.033404 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:00.033870 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.033525 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:20:00.289607 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.289527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:00.356449 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.356399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:00.785717 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.785684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:00.785912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.785737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:20:00.785912 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.785876 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:20:00.785912 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.785896 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:20:00.786068 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.785934 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:32.785920659 +0000 UTC m=+66.217066887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:20:00.786068 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.785975 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret podName:8246d4ad-44f1-4c46-8f31-62348f5a71d3 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:32.785941637 +0000 UTC m=+66.217087885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret") pod "global-pull-secret-syncer-vb66t" (UID: "8246d4ad-44f1-4c46-8f31-62348f5a71d3") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:20:00.887115 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:00.887080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:00.887291 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.887224 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:20:00.887291 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.887243 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:20:00.887291 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.887254 2564 projected.go:196] Error preparing data for projected volume kube-api-access-tm6kp for pod openshift-network-diagnostics/network-check-target-5x96h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:20:00.887452 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:00.887312 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp podName:bb0a6790-562a-48ef-a482-365aae792162 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:32.887293388 +0000 UTC m=+66.318439621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm6kp" (UniqueName: "kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp") pod "network-check-target-5x96h" (UID: "bb0a6790-562a-48ef-a482-365aae792162") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:20:01.036484 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:01.036415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:20:01.036484 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:01.036426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:01.036927 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:01.036525 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hc9s" podUID="a3311739-a779-4c3d-a848-5482e3cee472" Aug 05 17:20:01.036927 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:01.036632 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb66t" podUID="8246d4ad-44f1-4c46-8f31-62348f5a71d3" Aug 05 17:20:01.444668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:01.444634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:01.527005 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:01.526946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:01.950340 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:01.950297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:02.027802 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.027764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:02.032756 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.032738 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:02.032869 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:02.032835 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5x96h" podUID="bb0a6790-562a-48ef-a482-365aae792162" Aug 05 17:20:02.917568 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.917537 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-228.ec2.internal" event="NodeReady" Aug 05 17:20:02.917950 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.917676 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 17:20:02.963670 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.963637 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vnsvp"] Aug 05 17:20:02.966044 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.966019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:02.966437 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.966410 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p6xz5"] Aug 05 17:20:02.967981 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.967944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:02.972826 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.972797 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 17:20:02.972928 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.972907 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 17:20:02.974659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.973092 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6sjj\"" Aug 05 17:20:02.974659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.973214 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 17:20:02.974659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.973321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 17:20:02.974659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.973427 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 17:20:02.974659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.973565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rskhh\"" Aug 05 17:20:02.981246 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.981223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6xz5"] Aug 05 17:20:02.982270 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:02.982253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vnsvp"] Aug 05 17:20:03.036163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.036133 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:20:03.036328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.036310 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:03.038971 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.038923 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:20:03.039101 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.039045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rdxm\"" Aug 05 17:20:03.039157 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.039128 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:20:03.103270 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.103415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4367aa03-3c82-49a8-8111-a9f0cd2e124f-tmp-dir\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.103415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsz5\" (UniqueName: \"kubernetes.io/projected/4367aa03-3c82-49a8-8111-a9f0cd2e124f-kube-api-access-scsz5\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.103415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4367aa03-3c82-49a8-8111-a9f0cd2e124f-config-volume\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.103532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n2f\" (UniqueName: \"kubernetes.io/projected/8927124e-9aee-4387-b5f1-cbcd8b6782b4-kube-api-access-v2n2f\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.103532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.103465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8927124e-9aee-4387-b5f1-cbcd8b6782b4-cert\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.109625 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.109592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:03.204589 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8927124e-9aee-4387-b5f1-cbcd8b6782b4-cert\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4367aa03-3c82-49a8-8111-a9f0cd2e124f-tmp-dir\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsz5\" (UniqueName: \"kubernetes.io/projected/4367aa03-3c82-49a8-8111-a9f0cd2e124f-kube-api-access-scsz5\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4367aa03-3c82-49a8-8111-a9f0cd2e124f-config-volume\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:03.204723 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:03.204780 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.204759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n2f\" (UniqueName: \"kubernetes.io/projected/8927124e-9aee-4387-b5f1-cbcd8b6782b4-kube-api-access-v2n2f\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.205102 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:03.204797 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:03.704777829 +0000 UTC m=+37.135924075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:03.205102 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.205054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4367aa03-3c82-49a8-8111-a9f0cd2e124f-tmp-dir\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.205543 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.205519 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4367aa03-3c82-49a8-8111-a9f0cd2e124f-config-volume\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.209726 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.209706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8927124e-9aee-4387-b5f1-cbcd8b6782b4-cert\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.216570 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.216546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsz5\" (UniqueName: \"kubernetes.io/projected/4367aa03-3c82-49a8-8111-a9f0cd2e124f-kube-api-access-scsz5\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.217012 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.216989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n2f\" (UniqueName: \"kubernetes.io/projected/8927124e-9aee-4387-b5f1-cbcd8b6782b4-kube-api-access-v2n2f\") pod \"ingress-canary-p6xz5\" (UID: \"8927124e-9aee-4387-b5f1-cbcd8b6782b4\") " pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.285590 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.285508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:03.288244 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.288223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6xz5" Aug 05 17:20:03.431124 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.430991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6xz5"] Aug 05 17:20:03.434360 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:03.434331 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8927124e_9aee_4387_b5f1_cbcd8b6782b4.slice/crio-5c292cee7be26c481fb7012359ba567f97dad0fcbc8106bf2cbaa17fa4a95091 WatchSource:0}: Error finding container 5c292cee7be26c481fb7012359ba567f97dad0fcbc8106bf2cbaa17fa4a95091: Status 404 returned error can't find the container with id 5c292cee7be26c481fb7012359ba567f97dad0fcbc8106bf2cbaa17fa4a95091 Aug 05 17:20:03.710441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:03.710408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:03.710603 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:03.710510 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:03.710603 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:03.710560 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:04.710548262 +0000 UTC m=+38.141694494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:04.033691 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.033657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:04.036505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.036483 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 17:20:04.037510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.037492 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 17:20:04.037510 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.037504 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bmhgk\"" Aug 05 17:20:04.302074 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.302043 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="87c73aaeb718b63b33c414e0bc035764e2f89b58e1338256b9bf01b37a170ab4" exitCode=0 Aug 05 17:20:04.302251 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.302137 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"87c73aaeb718b63b33c414e0bc035764e2f89b58e1338256b9bf01b37a170ab4"} Aug 05 17:20:04.303503 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.303482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6xz5" event={"ID":"8927124e-9aee-4387-b5f1-cbcd8b6782b4","Type":"ContainerStarted","Data":"5c292cee7be26c481fb7012359ba567f97dad0fcbc8106bf2cbaa17fa4a95091"} Aug 05 17:20:04.717755 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:04.717678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:04.717882 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:04.717832 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:04.717985 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:04.717902 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:06.717880924 +0000 UTC m=+40.149027160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:05.309215 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:05.309169 2564 generic.go:356] "Generic (PLEG): container finished" podID="aaf06d87-8c29-4f6d-8c51-0d9834cab5ea" containerID="c39e3074d268c9d09a6c6a36d89b0041c9a6f1af7e7ddda48feaf5025e1ab00b" exitCode=0 Aug 05 17:20:05.309597 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:05.309225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerDied","Data":"c39e3074d268c9d09a6c6a36d89b0041c9a6f1af7e7ddda48feaf5025e1ab00b"} Aug 05 17:20:06.315493 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.315273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" event={"ID":"aaf06d87-8c29-4f6d-8c51-0d9834cab5ea","Type":"ContainerStarted","Data":"a0875932f64d2ae8565f454c77d8fdc178b07f0ae11086ce8981e94cee9c0c91"} Aug 05 17:20:06.316705 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.316677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6xz5" event={"ID":"8927124e-9aee-4387-b5f1-cbcd8b6782b4","Type":"ContainerStarted","Data":"e606bfc412ca43bc6b43b823e7d45e90ef1a9bbabd8e37dc16605ac61d6aa48c"} Aug 05 17:20:06.346261 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.346218 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8gsgs" podStartSLOduration=9.08548768 podStartE2EDuration="38.346204544s" totalImagesPullingTime="29.260716864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:06.343739526 +0000 UTC m=+39.774885794" watchObservedRunningTime="2026-08-05 17:20:06.346204544 +0000 UTC m=+39.777350793" Aug 05 17:20:06.368534 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.368479 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p6xz5" podStartSLOduration=1.756129464 podStartE2EDuration="4.368458303s" totalImagesPullingTime="2.612328839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:06.367903817 +0000 UTC m=+39.799050078" watchObservedRunningTime="2026-08-05 17:20:06.368458303 +0000 UTC m=+39.799604553" Aug 05 17:20:06.482270 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.482241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:06.581910 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.581829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:06.670175 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.670141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:06.730150 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:06.730119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:06.730274 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:06.730239 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:06.730321 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:06.730291 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:10.730277914 +0000 UTC m=+44.161424143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:09.428102 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:09.428060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:09.506430 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:09.506392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:10.373856 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:10.373824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:10.466876 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:10.466842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:10.758101 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:10.758017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:10.758251 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:10.758163 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:10.758251 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:10.758227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:18.758209995 +0000 UTC m=+52.189356224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:14.378555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:14.378518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:18.812374 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:18.812329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:18.812756 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:18.812508 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:18.812756 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:18.812576 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:20:34.812559941 +0000 UTC m=+68.243706169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:22.373639 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.373607 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn"] Aug 05 17:20:22.375986 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.375948 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z"] Aug 05 17:20:22.376113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.376064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.377488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.377470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" Aug 05 17:20:22.379906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.379888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn"] Aug 05 17:20:22.380508 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.380486 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 05 17:20:22.380587 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.380530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 05 17:20:22.381689 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.381664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:20:22.381773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.381700 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vkbbb\"" Aug 05 17:20:22.381773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.381741 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 05 17:20:22.381866 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.381837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-877mw\"" Aug 05 17:20:22.392918 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.392899 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z"] Aug 05 17:20:22.437698 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.437676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88pmk\" (UniqueName: \"kubernetes.io/projected/9fd352f0-ea50-4469-9e45-7f0fc0983a99-kube-api-access-88pmk\") pod \"network-check-source-65bdbbff45-4xp4z\" (UID: \"9fd352f0-ea50-4469-9e45-7f0fc0983a99\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" Aug 05 17:20:22.437815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.437742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/128344e9-2717-46e4-a3eb-6abcc088efc0-serving-cert\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.437815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.437773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwv6\" (UniqueName: \"kubernetes.io/projected/128344e9-2717-46e4-a3eb-6abcc088efc0-kube-api-access-mdwv6\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.437815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.437790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/128344e9-2717-46e4-a3eb-6abcc088efc0-config\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.469774 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.469753 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq"] Aug 05 17:20:22.471610 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.471592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.472555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.472537 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66d8d8f5b-rrnmh"] Aug 05 17:20:22.476758 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.476735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 17:20:22.476871 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.476827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 05 17:20:22.476871 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.476830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 17:20:22.476999 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.476900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.477765 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.477744 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:20:22.478144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.478122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4qrgf\"" Aug 05 17:20:22.478144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.478139 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 05 17:20:22.479642 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.479581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lfh76\"" Aug 05 17:20:22.479740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.479643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.480050 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480036 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 05 17:20:22.480138 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480062 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 17:20:22.480198 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480158 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 05 17:20:22.480198 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480166 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 05 17:20:22.480292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480254 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 17:20:22.480436 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.480421 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 05 17:20:22.483148 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.483133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 05 17:20:22.483454 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.483441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 05 17:20:22.483642 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.483629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 05 17:20:22.483736 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.483723 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nj4rs\"" Aug 05 17:20:22.489912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.489890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq"] Aug 05 17:20:22.491798 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.491779 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 05 17:20:22.492712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.492674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d8d8f5b-rrnmh"] Aug 05 17:20:22.498694 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.498676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:20:22.538485 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-default-certificate\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.538619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5hd\" (UniqueName: \"kubernetes.io/projected/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-kube-api-access-sp5hd\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.538619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjmx\" (UniqueName: \"kubernetes.io/projected/6462a8f5-6939-4711-9294-168ea9216841-kube-api-access-gzjmx\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.538619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.538619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/128344e9-2717-46e4-a3eb-6abcc088efc0-serving-cert\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwv6\" (UniqueName: \"kubernetes.io/projected/128344e9-2717-46e4-a3eb-6abcc088efc0-kube-api-access-mdwv6\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/128344e9-2717-46e4-a3eb-6abcc088efc0-config\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88pmk\" (UniqueName: \"kubernetes.io/projected/9fd352f0-ea50-4469-9e45-7f0fc0983a99-kube-api-access-88pmk\") pod \"network-check-source-65bdbbff45-4xp4z\" (UID: \"9fd352f0-ea50-4469-9e45-7f0fc0983a99\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.538808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkvm\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-stats-auth\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.539140 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.538927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.539319 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.539279 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/128344e9-2717-46e4-a3eb-6abcc088efc0-config\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.542578 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.542558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/128344e9-2717-46e4-a3eb-6abcc088efc0-serving-cert\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.548861 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.548836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwv6\" (UniqueName: \"kubernetes.io/projected/128344e9-2717-46e4-a3eb-6abcc088efc0-kube-api-access-mdwv6\") pod \"service-ca-operator-784f97bcf6-vtcnn\" (UID: \"128344e9-2717-46e4-a3eb-6abcc088efc0\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.549230 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.549210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88pmk\" (UniqueName: \"kubernetes.io/projected/9fd352f0-ea50-4469-9e45-7f0fc0983a99-kube-api-access-88pmk\") pod \"network-check-source-65bdbbff45-4xp4z\" (UID: \"9fd352f0-ea50-4469-9e45-7f0fc0983a99\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" Aug 05 17:20:22.639682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5hd\" (UniqueName: \"kubernetes.io/projected/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-kube-api-access-sp5hd\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.639682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjmx\" (UniqueName: \"kubernetes.io/projected/6462a8f5-6939-4711-9294-168ea9216841-kube-api-access-gzjmx\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.639682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.639843 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:22.639908 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.639906 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.639921 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:23.139900296 +0000 UTC m=+56.571046528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.639934 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9df757c4-96h9g: secret "image-registry-tls" not found Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.639844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkvm\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-stats-auth\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.640053 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls podName:fb294dab-5273-492e-b688-524ea8c3f6d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:23.140033665 +0000 UTC m=+56.571179907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls") pod "image-registry-5f9df757c4-96h9g" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7") : secret "image-registry-tls" not found Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.640255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-default-certificate\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.640788 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.640408 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:20:22.640788 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.640454 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:23.140439716 +0000 UTC m=+56.571585957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : secret "router-metrics-certs-default" not found Aug 05 17:20:22.640788 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.640788 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:22.640525 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:23.14051247 +0000 UTC m=+56.571658713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:22.640788 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.640667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.641212 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.641194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.641336 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.641312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.642500 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.642477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.642500 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.642493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.642724 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.642705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-stats-auth\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.642830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.642794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-default-certificate\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.655886 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.655855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.657112 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.657086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5hd\" (UniqueName: \"kubernetes.io/projected/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-kube-api-access-sp5hd\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:22.657254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.657234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjmx\" (UniqueName: \"kubernetes.io/projected/6462a8f5-6939-4711-9294-168ea9216841-kube-api-access-gzjmx\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:22.657314 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.657277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkvm\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:22.687057 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.687038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" Aug 05 17:20:22.691401 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.691384 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" Aug 05 17:20:22.694672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.694637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:22.759699 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.759667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:22.817545 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.817516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn"] Aug 05 17:20:22.820804 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:22.820781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128344e9_2717_46e4_a3eb_6abcc088efc0.slice/crio-24d129d1124972afd85f67e4520ae7750f1b1df93accd8c1dd426d3c7b50f846 WatchSource:0}: Error finding container 24d129d1124972afd85f67e4520ae7750f1b1df93accd8c1dd426d3c7b50f846: Status 404 returned error can't find the container with id 24d129d1124972afd85f67e4520ae7750f1b1df93accd8c1dd426d3c7b50f846 Aug 05 17:20:22.826532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.826502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:22.832961 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.832936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z"] Aug 05 17:20:22.836673 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:22.836652 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd352f0_ea50_4469_9e45_7f0fc0983a99.slice/crio-80fafb4585f6496f206c8ca6e9a6080becfd9b9794d5f7cced09cb1c3078c045 WatchSource:0}: Error finding container 80fafb4585f6496f206c8ca6e9a6080becfd9b9794d5f7cced09cb1c3078c045: Status 404 returned error can't find the container with id 80fafb4585f6496f206c8ca6e9a6080becfd9b9794d5f7cced09cb1c3078c045 Aug 05 17:20:22.891096 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:22.891018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:23.144473 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.144389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:23.144473 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.144464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.144488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.144502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144532 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144572 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144593 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144607 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9df757c4-96h9g: secret "image-registry-tls" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144598 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:24.14457905 +0000 UTC m=+57.575725280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144667 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:24.144652773 +0000 UTC m=+57.575799002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : secret "router-metrics-certs-default" not found Aug 05 17:20:23.144686 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144685 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:24.144673925 +0000 UTC m=+57.575820188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:23.144991 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:23.144700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls podName:fb294dab-5273-492e-b688-524ea8c3f6d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:24.1446924 +0000 UTC m=+57.575838628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls") pod "image-registry-5f9df757c4-96h9g" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7") : secret "image-registry-tls" not found Aug 05 17:20:23.356283 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.356247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" event={"ID":"128344e9-2717-46e4-a3eb-6abcc088efc0","Type":"ContainerStarted","Data":"24d129d1124972afd85f67e4520ae7750f1b1df93accd8c1dd426d3c7b50f846"} Aug 05 17:20:23.357249 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:23.357226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" event={"ID":"9fd352f0-ea50-4469-9e45-7f0fc0983a99","Type":"ContainerStarted","Data":"80fafb4585f6496f206c8ca6e9a6080becfd9b9794d5f7cced09cb1c3078c045"} Aug 05 17:20:24.154078 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.154030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:24.154514 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.154086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:24.154514 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.154109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:24.154514 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.154160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:24.154514 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154297 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:24.154514 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154364 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:26.154344579 +0000 UTC m=+59.585490807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:24.154799 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154781 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:20:24.154865 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154800 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9df757c4-96h9g: secret "image-registry-tls" not found Aug 05 17:20:24.154865 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls podName:fb294dab-5273-492e-b688-524ea8c3f6d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:26.15483126 +0000 UTC m=+59.585977506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls") pod "image-registry-5f9df757c4-96h9g" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7") : secret "image-registry-tls" not found Aug 05 17:20:24.154990 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154918 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:26.154907814 +0000 UTC m=+59.586054046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:24.155056 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.154996 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:20:24.155056 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:24.155039 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:26.155026436 +0000 UTC m=+59.586172683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : secret "router-metrics-certs-default" not found Aug 05 17:20:24.434690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.434601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:24.535093 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:24.535018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:25.364817 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:25.364777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" event={"ID":"128344e9-2717-46e4-a3eb-6abcc088efc0","Type":"ContainerStarted","Data":"2ba02baed86c02bfc3f0b18c94e9aa2f5d6c527b016cac229fb569845c0407c8"} Aug 05 17:20:25.382084 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:25.382029 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" podStartSLOduration=1.518266025 podStartE2EDuration="3.382008483s" totalImagesPullingTime="1.863742458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:25.381064686 +0000 UTC m=+58.812210957" watchObservedRunningTime="2026-08-05 17:20:25.382008483 +0000 UTC m=+58.813154733" Aug 05 17:20:25.437475 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:25.437354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:25.537306 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:25.537273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:25.963423 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:25.963391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.051849 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.051816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.173432 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.173401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:26.173590 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.173481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:26.173590 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.173507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:26.173590 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.173528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:26.173590 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173550 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173616 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.173599602 +0000 UTC m=+63.604745834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173617 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173622 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173658 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9df757c4-96h9g: secret "image-registry-tls" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173662 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.173642814 +0000 UTC m=+63.604789054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173721 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.173709868 +0000 UTC m=+63.604856097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : secret "router-metrics-certs-default" not found Aug 05 17:20:26.173746 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.173733 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls podName:fb294dab-5273-492e-b688-524ea8c3f6d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.173727858 +0000 UTC m=+63.604874086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls") pod "image-registry-5f9df757c4-96h9g" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7") : secret "image-registry-tls" not found Aug 05 17:20:26.257204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.257111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.348234 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.348204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.368431 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.368386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" event={"ID":"9fd352f0-ea50-4469-9e45-7f0fc0983a99","Type":"ContainerStarted","Data":"1fb74d36ab68cf5493a4e58eba9d2ef9aa8a153e496d4cce07f9fa48ca3894a4"} Aug 05 17:20:26.384913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.384727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4xp4z" podStartSLOduration=1.19715481 podStartE2EDuration="4.384708691s" totalImagesPullingTime="3.187553881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:26.38450152 +0000 UTC m=+59.815647781" watchObservedRunningTime="2026-08-05 17:20:26.384708691 +0000 UTC m=+59.815854939" Aug 05 17:20:26.708269 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.708233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.712856 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.712831 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wrl49"] Aug 05 17:20:26.714862 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.714844 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.717936 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.717915 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 17:20:26.718073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.718029 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 17:20:26.718131 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.718068 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfc7x\"" Aug 05 17:20:26.718732 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.718715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 17:20:26.718803 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.718722 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 17:20:26.746579 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.746552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrl49"] Aug 05 17:20:26.777724 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.777702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f8ff391-db3d-431f-bf38-be70bd1394ba-crio-socket\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.777863 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.777745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.777911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.777888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f8ff391-db3d-431f-bf38-be70bd1394ba-data-volume\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.777967 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.777918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpfp\" (UniqueName: \"kubernetes.io/projected/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-api-access-jcpfp\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.778018 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.777999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.863627 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.863600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:26.878368 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpfp\" (UniqueName: \"kubernetes.io/projected/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-api-access-jcpfp\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.878524 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.878524 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f8ff391-db3d-431f-bf38-be70bd1394ba-crio-socket\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.878524 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.878687 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8f8ff391-db3d-431f-bf38-be70bd1394ba-crio-socket\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.878740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.878726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f8ff391-db3d-431f-bf38-be70bd1394ba-data-volume\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.879137 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.879120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f8ff391-db3d-431f-bf38-be70bd1394ba-data-volume\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.881050 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.881034 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 17:20:26.882282 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.882259 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 17:20:26.888689 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.888669 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:26.888844 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.888736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:27.388715583 +0000 UTC m=+60.819861812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:26.888844 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.888740 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:20:26.888844 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:26.888794 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:27.388778209 +0000 UTC m=+60.819924472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : configmap "kube-rbac-proxy" not found Aug 05 17:20:26.895924 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.894601 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 17:20:26.905585 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.905564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 17:20:26.916048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.916024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpfp\" (UniqueName: \"kubernetes.io/projected/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-api-access-jcpfp\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:26.949245 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:26.949188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:27.483262 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:27.483223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:27.483648 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:27.483279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:27.483648 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:27.483342 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:20:27.483648 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:27.483373 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:27.483648 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:27.483385 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:28.483372122 +0000 UTC m=+61.914518350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : configmap "kube-rbac-proxy" not found Aug 05 17:20:27.483648 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:27.483510 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:28.483497908 +0000 UTC m=+61.914644136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:27.824229 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:27.824192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:28.303796 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:28.303773 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8r2f" Aug 05 17:20:28.492244 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:28.492209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:28.492604 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:28.492262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:28.492604 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:28.492354 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:28.492604 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:28.492388 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:20:28.492604 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:28.492431 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.492414615 +0000 UTC m=+63.923560855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:28.492604 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:28.492448 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:30.492441062 +0000 UTC m=+63.923587290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : configmap "kube-rbac-proxy" not found Aug 05 17:20:28.735978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:28.735878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:28.853152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:28.853129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9rcr_e6d3d3d4-1f92-48ad-8aeb-7bea6f342378/dns-node-resolver/0.log" Aug 05 17:20:29.853209 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:29.853184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8rkq_4e67ba62-36b0-4223-83f8-00eb8fbc53b9/node-ca/0.log" Aug 05 17:20:30.208104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.208023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.208125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.208159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208161 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208179 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9df757c4-96h9g: secret "image-registry-tls" not found Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.208189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:30.208257 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208239 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls podName:fb294dab-5273-492e-b688-524ea8c3f6d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:38.208224081 +0000 UTC m=+71.639370309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls") pod "image-registry-5f9df757c4-96h9g" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7") : secret "image-registry-tls" not found Aug 05 17:20:30.208479 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208277 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:20:30.208479 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208284 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:38.208269014 +0000 UTC m=+71.639415265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:30.208479 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208287 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:30.208479 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208313 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:38.208303032 +0000 UTC m=+71.639449260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : secret "router-metrics-certs-default" not found Aug 05 17:20:30.208479 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.208325 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:38.208317418 +0000 UTC m=+71.639463646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:30.253676 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.253648 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6xz5_8927124e-9aee-4387-b5f1-cbcd8b6782b4/serve-healthcheck-canary/0.log" Aug 05 17:20:30.510448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.510358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:30.510618 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.510477 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:20:30.510618 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:30.510500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:30.510618 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.510545 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:34.510526323 +0000 UTC m=+67.941672551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : configmap "kube-rbac-proxy" not found Aug 05 17:20:30.510739 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.510655 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:30.510739 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:30.510713 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:34.510700724 +0000 UTC m=+67.941846952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:31.501611 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:31.501577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.410992 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.410946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.511542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.511498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.644975 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.644928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.746506 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.746420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.814854 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.814830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:32.831803 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.831780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:20:32.831880 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.831841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:32.835088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.835071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:20:32.835180 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.835113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:20:32.842521 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:32.842498 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 05 17:20:32.842628 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:32.842552 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs podName:a3311739-a779-4c3d-a848-5482e3cee472 nodeName:}" failed. No retries permitted until 2026-08-05 17:21:36.842537986 +0000 UTC m=+130.273684215 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs") pod "network-metrics-daemon-2hc9s" (UID: "a3311739-a779-4c3d-a848-5482e3cee472") : secret "metrics-daemon-secret" not found Aug 05 17:20:32.844690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.844672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8246d4ad-44f1-4c46-8f31-62348f5a71d3-original-pull-secret\") pod \"global-pull-secret-syncer-vb66t\" (UID: \"8246d4ad-44f1-4c46-8f31-62348f5a71d3\") " pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:32.933048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.933017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:32.935536 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:32.935517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6kp\" (UniqueName: \"kubernetes.io/projected/bb0a6790-562a-48ef-a482-365aae792162-kube-api-access-tm6kp\") pod \"network-check-target-5x96h\" (UID: \"bb0a6790-562a-48ef-a482-365aae792162\") " pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:33.052343 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.052321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb66t" Aug 05 17:20:33.146670 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.146645 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bmhgk\"" Aug 05 17:20:33.153049 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.153017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:33.167207 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.167181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:33.181438 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.179504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb66t"] Aug 05 17:20:33.184110 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:33.184083 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8246d4ad_44f1_4c46_8f31_62348f5a71d3.slice/crio-35799ab41b650c714c1aee3de9ce7d357e7f54be6e772e0469d641d90001e3b6 WatchSource:0}: Error finding container 35799ab41b650c714c1aee3de9ce7d357e7f54be6e772e0469d641d90001e3b6: Status 404 returned error can't find the container with id 35799ab41b650c714c1aee3de9ce7d357e7f54be6e772e0469d641d90001e3b6 Aug 05 17:20:33.277975 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.277923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5x96h"] Aug 05 17:20:33.281639 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:33.281617 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0a6790_562a_48ef_a482_365aae792162.slice/crio-1a102ed2610a48e8b9508004239998e12f75754c38290035db881400b888bcaa WatchSource:0}: Error finding container 1a102ed2610a48e8b9508004239998e12f75754c38290035db881400b888bcaa: Status 404 returned error can't find the container with id 1a102ed2610a48e8b9508004239998e12f75754c38290035db881400b888bcaa Aug 05 17:20:33.388871 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.388836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5x96h" event={"ID":"bb0a6790-562a-48ef-a482-365aae792162","Type":"ContainerStarted","Data":"08b73882ca93e7b531cf3c5e462a452f32a0450a9f7ccbf78e43fdd2a68b4f14"} Aug 05 17:20:33.389034 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.388895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5x96h" event={"ID":"bb0a6790-562a-48ef-a482-365aae792162","Type":"ContainerStarted","Data":"1a102ed2610a48e8b9508004239998e12f75754c38290035db881400b888bcaa"} Aug 05 17:20:33.389034 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.388964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:20:33.390001 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.389978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vb66t" event={"ID":"8246d4ad-44f1-4c46-8f31-62348f5a71d3","Type":"ContainerStarted","Data":"35799ab41b650c714c1aee3de9ce7d357e7f54be6e772e0469d641d90001e3b6"} Aug 05 17:20:33.409149 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:33.409105 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5x96h" podStartSLOduration=65.40909086 podStartE2EDuration="1m5.40909086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:33.408706583 +0000 UTC m=+66.839852836" watchObservedRunningTime="2026-08-05 17:20:33.40909086 +0000 UTC m=+66.840237091" Aug 05 17:20:34.121225 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.121194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:34.347303 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.347273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:34.505188 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.505098 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:34.547152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.547117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:34.547324 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.547197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:34.547324 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.547236 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:34.547324 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.547307 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:42.547289609 +0000 UTC m=+75.978435841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:34.547324 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.547307 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:20:34.547534 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.547361 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:42.547346953 +0000 UTC m=+75.978493195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : configmap "kube-rbac-proxy" not found Aug 05 17:20:34.851022 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.850987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:20:34.851216 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.851158 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:20:34.851262 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:34.851226 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls podName:4367aa03-3c82-49a8-8111-a9f0cd2e124f nodeName:}" failed. No retries permitted until 2026-08-05 17:21:06.851210766 +0000 UTC m=+100.282356998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls") pod "dns-default-vnsvp" (UID: "4367aa03-3c82-49a8-8111-a9f0cd2e124f") : secret "dns-default-metrics-tls" not found Aug 05 17:20:34.909527 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:34.909492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:35.712106 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:35.712059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.008923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.008843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.109848 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.109813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.168183 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.168152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.301424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.301339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.508423 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.508370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.709293 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.709200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:36.783058 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:36.783015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:37.712516 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:37.712475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:38.114892 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.114863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:38.277604 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.277575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.277609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.277636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.277662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:38.277740 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:38.277757 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle podName:6462a8f5-6939-4711-9294-168ea9216841 nodeName:}" failed. No retries permitted until 2026-08-05 17:20:54.277735845 +0000 UTC m=+87.708882088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle") pod "router-default-66d8d8f5b-rrnmh" (UID: "6462a8f5-6939-4711-9294-168ea9216841") : configmap references non-existent config key: service-ca.crt Aug 05 17:20:38.277783 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:38.277783 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls podName:e1e319f2-7d5e-4991-9c24-713a8d52ce1d nodeName:}" failed. No retries permitted until 2026-08-05 17:20:54.277773194 +0000 UTC m=+87.708919425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-p8pcq" (UID: "e1e319f2-7d5e-4991-9c24-713a8d52ce1d") : secret "cluster-monitoring-operator-tls" not found Aug 05 17:20:38.280191 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.280170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6462a8f5-6939-4711-9294-168ea9216841-metrics-certs\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:38.280294 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.280194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"image-registry-5f9df757c4-96h9g\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:38.312085 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.312055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:38.398826 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.398762 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nj4rs\"" Aug 05 17:20:38.404913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.404891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vb66t" event={"ID":"8246d4ad-44f1-4c46-8f31-62348f5a71d3","Type":"ContainerStarted","Data":"c99afa4c525ef024c8ae37e23a8a794c9aa246f25d9ed01b8a5f463f91188de3"} Aug 05 17:20:38.407195 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.407183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:38.424938 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.424891 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vb66t" podStartSLOduration=66.320502762 podStartE2EDuration="1m10.424874069s" totalImagesPullingTime="4.104371307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:38.424485233 +0000 UTC m=+71.855631501" watchObservedRunningTime="2026-08-05 17:20:38.424874069 +0000 UTC m=+71.856020320" Aug 05 17:20:38.513207 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.513179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:38.528304 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.528190 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:20:38.530522 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:38.530497 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb294dab_5273_492e_b688_524ea8c3f6d7.slice/crio-0bc85b36f1374fe07ce1b2b80013e6a0ede8b902febd9b73cc23be05a0b58323 WatchSource:0}: Error finding container 0bc85b36f1374fe07ce1b2b80013e6a0ede8b902febd9b73cc23be05a0b58323: Status 404 returned error can't find the container with id 0bc85b36f1374fe07ce1b2b80013e6a0ede8b902febd9b73cc23be05a0b58323 Aug 05 17:20:38.624993 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.624930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:38.730347 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:38.730260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:39.133292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.133253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:39.409741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.409656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" event={"ID":"fb294dab-5273-492e-b688-524ea8c3f6d7","Type":"ContainerStarted","Data":"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce"} Aug 05 17:20:39.409741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.409707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" event={"ID":"fb294dab-5273-492e-b688-524ea8c3f6d7","Type":"ContainerStarted","Data":"0bc85b36f1374fe07ce1b2b80013e6a0ede8b902febd9b73cc23be05a0b58323"} Aug 05 17:20:39.430380 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.430334 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" podStartSLOduration=17.430319404 podStartE2EDuration="17.430319404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:39.42877945 +0000 UTC m=+72.859925711" watchObservedRunningTime="2026-08-05 17:20:39.430319404 +0000 UTC m=+72.861465654" Aug 05 17:20:39.533485 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.533447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:39.728104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:39.728026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:40.112783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:40.112754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:40.326073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:40.326043 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:40.415280 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:40.415207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:40.509122 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:40.509092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:40.711319 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:40.711238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:41.710770 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:41.710736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:41.911264 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:41.911231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:42.347271 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.347233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:42.530519 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.530485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:42.615095 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.615009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:42.615095 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.615073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:42.615300 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:42.615148 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:20:42.615300 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:20:42.615225 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls podName:8f8ff391-db3d-431f-bf38-be70bd1394ba nodeName:}" failed. No retries permitted until 2026-08-05 17:20:58.615207933 +0000 UTC m=+92.046354161 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrl49" (UID: "8f8ff391-db3d-431f-bf38-be70bd1394ba") : secret "insights-runtime-extractor-tls" not found Aug 05 17:20:42.615595 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.615575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8f8ff391-db3d-431f-bf38-be70bd1394ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:42.704637 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:42.704607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:43.122024 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:43.121991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:45.287430 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:45.287399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:45.356250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:45.356212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:49.192065 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.191945 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:20:49.246234 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.246204 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dd84f8cf4-r96wh"] Aug 05 17:20:49.248055 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.248039 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.269968 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.269926 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd84f8cf4-r96wh"] Aug 05 17:20:49.280915 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.280894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:49.371115 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-tls\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-trusted-ca\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1130020e-8f63-445f-9ba9-eb50c9c91b10-ca-trust-extracted\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-certificates\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-bound-sa-token\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-image-registry-private-configuration\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-installation-pull-secrets\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.371352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.371333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb7l\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-kube-api-access-grb7l\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.383574 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.383545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:49.472492 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-image-registry-private-configuration\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472492 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-installation-pull-secrets\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472492 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7l\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-kube-api-access-grb7l\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-tls\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-trusted-ca\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1130020e-8f63-445f-9ba9-eb50c9c91b10-ca-trust-extracted\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-certificates\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.472741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.472692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-bound-sa-token\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.473112 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.473089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1130020e-8f63-445f-9ba9-eb50c9c91b10-ca-trust-extracted\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.473569 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.473511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-certificates\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.473947 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.473920 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1130020e-8f63-445f-9ba9-eb50c9c91b10-trusted-ca\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.474981 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.474939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-image-registry-private-configuration\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.475152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.475131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-registry-tls\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.475234 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.475221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1130020e-8f63-445f-9ba9-eb50c9c91b10-installation-pull-secrets\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.483008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.482988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb7l\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-kube-api-access-grb7l\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.483440 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.483421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1130020e-8f63-445f-9ba9-eb50c9c91b10-bound-sa-token\") pod \"image-registry-7dd84f8cf4-r96wh\" (UID: \"1130020e-8f63-445f-9ba9-eb50c9c91b10\") " pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.555730 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.555707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:49.681805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:49.681783 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd84f8cf4-r96wh"] Aug 05 17:20:49.684002 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:49.683951 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1130020e_8f63_445f_9ba9_eb50c9c91b10.slice/crio-f7f44cb998035a6b1bc294a2c5b507d6c94938d900088c10450d34bf7c0c0a82 WatchSource:0}: Error finding container f7f44cb998035a6b1bc294a2c5b507d6c94938d900088c10450d34bf7c0c0a82: Status 404 returned error can't find the container with id f7f44cb998035a6b1bc294a2c5b507d6c94938d900088c10450d34bf7c0c0a82 Aug 05 17:20:50.445588 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:50.445554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" event={"ID":"1130020e-8f63-445f-9ba9-eb50c9c91b10","Type":"ContainerStarted","Data":"6f0665152ecca8a298db86f06c7e9b29eac5e26146f3ba113264f4a040315381"} Aug 05 17:20:50.446079 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:50.445607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" event={"ID":"1130020e-8f63-445f-9ba9-eb50c9c91b10","Type":"ContainerStarted","Data":"f7f44cb998035a6b1bc294a2c5b507d6c94938d900088c10450d34bf7c0c0a82"} Aug 05 17:20:50.446079 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:50.445648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:20:50.470218 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:50.470178 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" podStartSLOduration=1.470164428 podStartE2EDuration="1.470164428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:50.468620644 +0000 UTC m=+83.899766885" watchObservedRunningTime="2026-08-05 17:20:50.470164428 +0000 UTC m=+83.901310676" Aug 05 17:20:54.311620 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.311576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:54.311620 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.311635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:54.312394 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.312369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6462a8f5-6939-4711-9294-168ea9216841-service-ca-bundle\") pod \"router-default-66d8d8f5b-rrnmh\" (UID: \"6462a8f5-6939-4711-9294-168ea9216841\") " pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:54.314053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.314036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1e319f2-7d5e-4991-9c24-713a8d52ce1d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-p8pcq\" (UID: \"e1e319f2-7d5e-4991-9c24-713a8d52ce1d\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:54.587277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.587201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4qrgf\"" Aug 05 17:20:54.594079 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.594057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lfh76\"" Aug 05 17:20:54.595098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.595083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" Aug 05 17:20:54.602771 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.602750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:54.728187 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.728150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d8d8f5b-rrnmh"] Aug 05 17:20:54.731341 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:54.731310 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462a8f5_6939_4711_9294_168ea9216841.slice/crio-832142adb33caeec3f5d929ce9bccbf65e42250112b633a4dfaacb9dd86b8d46 WatchSource:0}: Error finding container 832142adb33caeec3f5d929ce9bccbf65e42250112b633a4dfaacb9dd86b8d46: Status 404 returned error can't find the container with id 832142adb33caeec3f5d929ce9bccbf65e42250112b633a4dfaacb9dd86b8d46 Aug 05 17:20:54.740736 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:54.740696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq"] Aug 05 17:20:54.743063 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:54.743040 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e319f2_7d5e_4991_9c24_713a8d52ce1d.slice/crio-6d61540fcf8f178fc1b391c2a02eed8de42cbaac069ca1c445f9bf8c1329c0fe WatchSource:0}: Error finding container 6d61540fcf8f178fc1b391c2a02eed8de42cbaac069ca1c445f9bf8c1329c0fe: Status 404 returned error can't find the container with id 6d61540fcf8f178fc1b391c2a02eed8de42cbaac069ca1c445f9bf8c1329c0fe Aug 05 17:20:55.463029 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.462996 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" event={"ID":"6462a8f5-6939-4711-9294-168ea9216841","Type":"ContainerStarted","Data":"812ea6111dc0844e5532a9736823727ec5af48ff245e30a80bb924208c8dba07"} Aug 05 17:20:55.463463 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.463049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" event={"ID":"6462a8f5-6939-4711-9294-168ea9216841","Type":"ContainerStarted","Data":"832142adb33caeec3f5d929ce9bccbf65e42250112b633a4dfaacb9dd86b8d46"} Aug 05 17:20:55.464501 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.464453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" event={"ID":"e1e319f2-7d5e-4991-9c24-713a8d52ce1d","Type":"ContainerStarted","Data":"6d61540fcf8f178fc1b391c2a02eed8de42cbaac069ca1c445f9bf8c1329c0fe"} Aug 05 17:20:55.486886 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.486844 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" podStartSLOduration=33.486825153 podStartE2EDuration="33.486825153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:55.486524724 +0000 UTC m=+88.917670974" watchObservedRunningTime="2026-08-05 17:20:55.486825153 +0000 UTC m=+88.917971404" Aug 05 17:20:55.603214 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.603178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:55.606003 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:55.605981 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:56.469033 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.469002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" event={"ID":"e1e319f2-7d5e-4991-9c24-713a8d52ce1d","Type":"ContainerStarted","Data":"cc050842cd49d26a2544b6dd3648acf1824a2e6ea4c29fb712b8635ccff33223"} Aug 05 17:20:56.469507 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.469298 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:56.470462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.470442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66d8d8f5b-rrnmh" Aug 05 17:20:56.490798 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.490745 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-p8pcq" podStartSLOduration=32.975013742 podStartE2EDuration="34.490725987s" totalImagesPullingTime="1.515712245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:56.488509261 +0000 UTC m=+89.919655517" watchObservedRunningTime="2026-08-05 17:20:56.490725987 +0000 UTC m=+89.921872238" Aug 05 17:20:56.596778 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.596750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:56.695497 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.695466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:56.777069 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:56.776936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.355062 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.355023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.561913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.561876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.661398 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.661311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.733379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.733349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.920625 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.920537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.998735 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:57.998695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:58.643819 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:58.643786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:58.646260 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:58.646238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8f8ff391-db3d-431f-bf38-be70bd1394ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrl49\" (UID: \"8f8ff391-db3d-431f-bf38-be70bd1394ba\") " pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:58.826193 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:58.826163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfc7x\"" Aug 05 17:20:58.833811 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:58.833791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrl49" Aug 05 17:20:58.993922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:58.993902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrl49"] Aug 05 17:20:58.996315 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:20:58.996288 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8ff391_db3d_431f_bf38_be70bd1394ba.slice/crio-419ee3f5b55a62ed0eeae632e67824896e21ffd9390c875e96c5069a94998f93 WatchSource:0}: Error finding container 419ee3f5b55a62ed0eeae632e67824896e21ffd9390c875e96c5069a94998f93: Status 404 returned error can't find the container with id 419ee3f5b55a62ed0eeae632e67824896e21ffd9390c875e96c5069a94998f93 Aug 05 17:20:59.197097 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.197023 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:20:59.479883 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.479807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrl49" event={"ID":"8f8ff391-db3d-431f-bf38-be70bd1394ba","Type":"ContainerStarted","Data":"2ac7a6a83dadb734a27af90fd39f17f1acdba5b14e62a8ef985d2c3eae74c413"} Aug 05 17:20:59.479883 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.479858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrl49" event={"ID":"8f8ff391-db3d-431f-bf38-be70bd1394ba","Type":"ContainerStarted","Data":"419ee3f5b55a62ed0eeae632e67824896e21ffd9390c875e96c5069a94998f93"} Aug 05 17:20:59.787046 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.786920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:59.828465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.828431 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-wrc2l"] Aug 05 17:20:59.844253 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.844233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:20:59.853154 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.853133 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-wrc2l"] Aug 05 17:20:59.860589 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.860571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 05 17:20:59.861092 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.861075 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qwwnk\"" Aug 05 17:20:59.861092 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.861085 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 05 17:20:59.862258 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.862245 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 17:20:59.892744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.892715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:59.953927 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.953884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2r7s\" (UniqueName: \"kubernetes.io/projected/d723d786-ee1b-44d9-b406-35a7b61e9bb6-kube-api-access-z2r7s\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:20:59.953927 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.953930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:20:59.954113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.953986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:20:59.954113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:20:59.954046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d723d786-ee1b-44d9-b406-35a7b61e9bb6-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.054624 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.054595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.054773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.054648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d723d786-ee1b-44d9-b406-35a7b61e9bb6-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.054773 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:00.054737 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 05 17:21:00.054836 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:00.054813 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls podName:d723d786-ee1b-44d9-b406-35a7b61e9bb6 nodeName:}" failed. No retries permitted until 2026-08-05 17:21:00.554797297 +0000 UTC m=+93.985943525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-wrc2l" (UID: "d723d786-ee1b-44d9-b406-35a7b61e9bb6") : secret "prometheus-operator-tls" not found Aug 05 17:21:00.054836 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.054830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2r7s\" (UniqueName: \"kubernetes.io/projected/d723d786-ee1b-44d9-b406-35a7b61e9bb6-kube-api-access-z2r7s\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.054908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.054862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.055241 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.055226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d723d786-ee1b-44d9-b406-35a7b61e9bb6-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.057218 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.057200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.073083 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.073059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2r7s\" (UniqueName: \"kubernetes.io/projected/d723d786-ee1b-44d9-b406-35a7b61e9bb6-kube-api-access-z2r7s\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.161209 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.161172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:00.484695 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.484656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrl49" event={"ID":"8f8ff391-db3d-431f-bf38-be70bd1394ba","Type":"ContainerStarted","Data":"7091460d87af51733060900d72bfe1379dff471df4ec19c39898193399d5220a"} Aug 05 17:21:00.517445 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.517418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:00.558479 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.558456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.560817 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.560787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d723d786-ee1b-44d9-b406-35a7b61e9bb6-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-wrc2l\" (UID: \"d723d786-ee1b-44d9-b406-35a7b61e9bb6\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.593109 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.593072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:00.752049 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.751932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" Aug 05 17:21:00.872343 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:00.872315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-wrc2l"] Aug 05 17:21:00.874941 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:00.874912 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd723d786_ee1b_44d9_b406_35a7b61e9bb6.slice/crio-b4b6f15b52c1f88c95684d938c017c1fed08ad26638eafac9472c8d242992c81 WatchSource:0}: Error finding container b4b6f15b52c1f88c95684d938c017c1fed08ad26638eafac9472c8d242992c81: Status 404 returned error can't find the container with id b4b6f15b52c1f88c95684d938c017c1fed08ad26638eafac9472c8d242992c81 Aug 05 17:21:01.490260 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:01.490228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" event={"ID":"d723d786-ee1b-44d9-b406-35a7b61e9bb6","Type":"ContainerStarted","Data":"b4b6f15b52c1f88c95684d938c017c1fed08ad26638eafac9472c8d242992c81"} Aug 05 17:21:02.392501 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.392405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:02.495112 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.495082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrl49" event={"ID":"8f8ff391-db3d-431f-bf38-be70bd1394ba","Type":"ContainerStarted","Data":"5000dffc8a2de5b0ecb36c7eceee28642d5e4ada4e735707b945ece16dce5584"} Aug 05 17:21:02.496783 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.496760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" event={"ID":"d723d786-ee1b-44d9-b406-35a7b61e9bb6","Type":"ContainerStarted","Data":"6aa65dbe1c8091e363d0043d485c3c7260c5b0b46b9bad783a42fc5ee24444c7"} Aug 05 17:21:02.496882 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.496805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" event={"ID":"d723d786-ee1b-44d9-b406-35a7b61e9bb6","Type":"ContainerStarted","Data":"117b46db21c1b83c0a596a67a13718e71d2d807a9a84c13cd2a9691954d34557"} Aug 05 17:21:02.516557 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.516511 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wrl49" podStartSLOduration=33.602725966 podStartE2EDuration="36.516496357s" totalImagesPullingTime="2.913770391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:20:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:02.515488979 +0000 UTC m=+95.946635243" watchObservedRunningTime="2026-08-05 17:21:02.516496357 +0000 UTC m=+95.947642601" Aug 05 17:21:02.533694 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:02.533653 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-wrc2l" podStartSLOduration=2.308779596 podStartE2EDuration="3.533639493s" totalImagesPullingTime="1.224859897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:02.53262527 +0000 UTC m=+95.963771519" watchObservedRunningTime="2026-08-05 17:21:02.533639493 +0000 UTC m=+95.964785742" Aug 05 17:21:03.566191 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:03.566153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:03.634367 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:03.634324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:03.915204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:03.915120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.144913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.144879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.172118 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.172049 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqwzl"] Aug 05 17:21:04.174258 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.174241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.178108 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.178091 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 17:21:04.178344 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.178330 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 17:21:04.178444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.178402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kbf9b\"" Aug 05 17:21:04.178631 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.178611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 17:21:04.191309 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-metrics-client-ca\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-root\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191544 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-textfile\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191544 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-sys\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191544 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191646 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxtc\" (UniqueName: \"kubernetes.io/projected/11f51db9-c061-4439-b97a-6dc61e8f53b7-kube-api-access-kfxtc\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.191646 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.191587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-wtmp\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.248397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.248356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.292511 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-root\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-textfile\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-sys\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-root\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxtc\" (UniqueName: \"kubernetes.io/projected/11f51db9-c061-4439-b97a-6dc61e8f53b7-kube-api-access-kfxtc\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292657 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-sys\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-wtmp\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-metrics-client-ca\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:04.292817 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-textfile\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.292840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-wtmp\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.292922 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:04.292886 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls podName:11f51db9-c061-4439-b97a-6dc61e8f53b7 nodeName:}" failed. No retries permitted until 2026-08-05 17:21:04.792866049 +0000 UTC m=+98.224012278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls") pod "node-exporter-gqwzl" (UID: "11f51db9-c061-4439-b97a-6dc61e8f53b7") : secret "node-exporter-tls" not found Aug 05 17:21:04.293307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.293283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.293405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.293390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11f51db9-c061-4439-b97a-6dc61e8f53b7-metrics-client-ca\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.295090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.295071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.305139 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.305118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxtc\" (UniqueName: \"kubernetes.io/projected/11f51db9-c061-4439-b97a-6dc61e8f53b7-kube-api-access-kfxtc\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.343347 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.343321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.394403 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.394382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5x96h" Aug 05 17:21:04.499561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.499475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.656076 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.656039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.757881 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.757795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.800047 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.800021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.802320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.802281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11f51db9-c061-4439-b97a-6dc61e8f53b7-node-exporter-tls\") pod \"node-exporter-gqwzl\" (UID: \"11f51db9-c061-4439-b97a-6dc61e8f53b7\") " pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:04.859678 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.859650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.958374 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:04.958333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:05.083394 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.083367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqwzl" Aug 05 17:21:05.094279 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:05.094253 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f51db9_c061_4439_b97a_6dc61e8f53b7.slice/crio-2dcd812062e32e0f9033d7abd943c1ec2a0f2ad270bfc654fbee5c15a5d5156d WatchSource:0}: Error finding container 2dcd812062e32e0f9033d7abd943c1ec2a0f2ad270bfc654fbee5c15a5d5156d: Status 404 returned error can't find the container with id 2dcd812062e32e0f9033d7abd943c1ec2a0f2ad270bfc654fbee5c15a5d5156d Aug 05 17:21:05.174300 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.174250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:05.256210 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.256179 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:21:05.258872 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.258846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.260938 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.260909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:05.261580 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.261562 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 05 17:21:05.261679 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.261563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 05 17:21:05.262042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 05 17:21:05.262135 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262102 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 05 17:21:05.262190 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262170 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 05 17:21:05.262488 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rdkj5\"" Aug 05 17:21:05.262567 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262508 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 05 17:21:05.262622 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 05 17:21:05.262941 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262924 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 05 17:21:05.263047 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.262929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 05 17:21:05.274363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.274340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:21:05.304248 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304248 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vzl\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.304608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.304601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.379163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.379073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:05.405203 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405316 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405494 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405539 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405582 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405854 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405984 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405984 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.405984 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.405968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.406165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.406165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.406165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.406165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vzl\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.406353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.407306 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.406655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.407306 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:05.406699 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle podName:288ebc5a-20d7-4c0d-996e-3573ebcf06f5 nodeName:}" failed. No retries permitted until 2026-08-05 17:21:05.906671005 +0000 UTC m=+99.337817237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5") : configmap references non-existent config key: ca-bundle.crt Aug 05 17:21:05.408529 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.408504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.408690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.408669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.408769 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.408722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.409084 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.409057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.409168 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.409062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.409705 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.409684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.410276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.410261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.410276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.410278 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.410926 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.410913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.415827 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.415804 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vzl\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.506633 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.506600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqwzl" event={"ID":"11f51db9-c061-4439-b97a-6dc61e8f53b7","Type":"ContainerStarted","Data":"2dcd812062e32e0f9033d7abd943c1ec2a0f2ad270bfc654fbee5c15a5d5156d"} Aug 05 17:21:05.909854 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.909813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:05.911016 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:05.910987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:06.171113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.171043 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:21:06.381742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.381696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:21:06.383716 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:06.383694 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288ebc5a_20d7_4c0d_996e_3573ebcf06f5.slice/crio-d9a9e81a96feac130498fc4be2c1c31a9a37a26c9ed309febda20296cc81c6cd WatchSource:0}: Error finding container d9a9e81a96feac130498fc4be2c1c31a9a37a26c9ed309febda20296cc81c6cd: Status 404 returned error can't find the container with id d9a9e81a96feac130498fc4be2c1c31a9a37a26c9ed309febda20296cc81c6cd Aug 05 17:21:06.511691 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.511659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"d9a9e81a96feac130498fc4be2c1c31a9a37a26c9ed309febda20296cc81c6cd"} Aug 05 17:21:06.513293 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.513262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqwzl" event={"ID":"11f51db9-c061-4439-b97a-6dc61e8f53b7","Type":"ContainerStarted","Data":"09c74c634c759bfa5a6f1be8317df87679aebcba2142eb7c6db0db7f1af585aa"} Aug 05 17:21:06.919479 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.919443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:21:06.922143 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:06.922116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4367aa03-3c82-49a8-8111-a9f0cd2e124f-metrics-tls\") pod \"dns-default-vnsvp\" (UID: \"4367aa03-3c82-49a8-8111-a9f0cd2e124f\") " pod="openshift-dns/dns-default-vnsvp" Aug 05 17:21:07.185448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.185376 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6sjj\"" Aug 05 17:21:07.193019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.192994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vnsvp" Aug 05 17:21:07.205182 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.205152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:07.307438 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.307376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:07.475368 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.475339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:07.489741 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.489713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vnsvp"] Aug 05 17:21:07.492691 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:07.492666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4367aa03_3c82_49a8_8111_a9f0cd2e124f.slice/crio-6d34905facbd28682a5d0016c6d0ab4b812b4ce5d76460be42a5abc4c866ee49 WatchSource:0}: Error finding container 6d34905facbd28682a5d0016c6d0ab4b812b4ce5d76460be42a5abc4c866ee49: Status 404 returned error can't find the container with id 6d34905facbd28682a5d0016c6d0ab4b812b4ce5d76460be42a5abc4c866ee49 Aug 05 17:21:07.517823 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.517800 2564 generic.go:356] "Generic (PLEG): container finished" podID="11f51db9-c061-4439-b97a-6dc61e8f53b7" containerID="09c74c634c759bfa5a6f1be8317df87679aebcba2142eb7c6db0db7f1af585aa" exitCode=0 Aug 05 17:21:07.517939 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.517866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqwzl" event={"ID":"11f51db9-c061-4439-b97a-6dc61e8f53b7","Type":"ContainerDied","Data":"09c74c634c759bfa5a6f1be8317df87679aebcba2142eb7c6db0db7f1af585aa"} Aug 05 17:21:07.519093 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.519072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnsvp" event={"ID":"4367aa03-3c82-49a8-8111-a9f0cd2e124f","Type":"ContainerStarted","Data":"6d34905facbd28682a5d0016c6d0ab4b812b4ce5d76460be42a5abc4c866ee49"} Aug 05 17:21:07.520686 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.520664 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="b3791d7a000fcd751a12bb1a138e38f6a852d305d26ead4611fcc9782e77fbfe" exitCode=0 Aug 05 17:21:07.520769 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.520736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"b3791d7a000fcd751a12bb1a138e38f6a852d305d26ead4611fcc9782e77fbfe"} Aug 05 17:21:07.586151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:07.586121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:08.527321 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:08.527279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqwzl" event={"ID":"11f51db9-c061-4439-b97a-6dc61e8f53b7","Type":"ContainerStarted","Data":"7ef9336ce46e9e03338a6cd06d530063ba9144f21d771fe0b33ffe1dbf8ffc7b"} Aug 05 17:21:08.527725 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:08.527345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqwzl" event={"ID":"11f51db9-c061-4439-b97a-6dc61e8f53b7","Type":"ContainerStarted","Data":"ee3114d0db0b280baee8c1a4415f8eabd8e9191a56289a89e792cca1d690b8f0"} Aug 05 17:21:08.550172 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:08.550109 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqwzl" podStartSLOduration=3.213674147 podStartE2EDuration="4.550093012s" totalImagesPullingTime="1.267655399s" totalInitContainerRuntime="68.763466ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:08.548390564 +0000 UTC m=+101.979536814" watchObservedRunningTime="2026-08-05 17:21:08.550093012 +0000 UTC m=+101.981239261" Aug 05 17:21:09.019013 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.018949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.027203 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.027179 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv"] Aug 05 17:21:09.030389 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.030364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.032773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.032752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 05 17:21:09.032925 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.032758 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jkwbl\"" Aug 05 17:21:09.040857 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.040834 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv"] Aug 05 17:21:09.139985 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.139937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7de704b1-f978-4db8-9043-9e37630d4aa4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.140182 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.139992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsz6\" (UniqueName: \"kubernetes.io/projected/7de704b1-f978-4db8-9043-9e37630d4aa4-kube-api-access-glsz6\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.240826 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.240792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7de704b1-f978-4db8-9043-9e37630d4aa4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.240997 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.240834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsz6\" (UniqueName: \"kubernetes.io/projected/7de704b1-f978-4db8-9043-9e37630d4aa4-kube-api-access-glsz6\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.243516 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.243486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7de704b1-f978-4db8-9043-9e37630d4aa4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.255361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.255339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsz6\" (UniqueName: \"kubernetes.io/projected/7de704b1-f978-4db8-9043-9e37630d4aa4-kube-api-access-glsz6\") pod \"monitoring-plugin-6dffbbd67-s87fv\" (UID: \"7de704b1-f978-4db8-9043-9e37630d4aa4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.273831 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.273773 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-l9kqb"] Aug 05 17:21:09.276386 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.276369 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:09.279539 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.279510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gzdbx\"" Aug 05 17:21:09.279539 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.279531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 05 17:21:09.280036 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.279942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 05 17:21:09.302797 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.302772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.310074 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.310051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-l9kqb"] Aug 05 17:21:09.341814 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.341790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klb6\" (UniqueName: \"kubernetes.io/projected/b64bd4b2-fbd2-41c2-beb8-9da5351adb1f-kube-api-access-5klb6\") pod \"downloads-77c974ffcf-l9kqb\" (UID: \"b64bd4b2-fbd2-41c2-beb8-9da5351adb1f\") " pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:09.342575 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.342553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:09.432899 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.432836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.443398 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.443330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klb6\" (UniqueName: \"kubernetes.io/projected/b64bd4b2-fbd2-41c2-beb8-9da5351adb1f-kube-api-access-5klb6\") pod \"downloads-77c974ffcf-l9kqb\" (UID: \"b64bd4b2-fbd2-41c2-beb8-9da5351adb1f\") " pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:09.463394 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.463361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klb6\" (UniqueName: \"kubernetes.io/projected/b64bd4b2-fbd2-41c2-beb8-9da5351adb1f-kube-api-access-5klb6\") pod \"downloads-77c974ffcf-l9kqb\" (UID: \"b64bd4b2-fbd2-41c2-beb8-9da5351adb1f\") " pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:09.520888 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.520856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv"] Aug 05 17:21:09.529424 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:09.529395 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de704b1_f978_4db8_9043_9e37630d4aa4.slice/crio-3351c16fb8baad9a4fd0521d78c8f7bfdc313cf0eb919f843caad0705e98c391 WatchSource:0}: Error finding container 3351c16fb8baad9a4fd0521d78c8f7bfdc313cf0eb919f843caad0705e98c391: Status 404 returned error can't find the container with id 3351c16fb8baad9a4fd0521d78c8f7bfdc313cf0eb919f843caad0705e98c391 Aug 05 17:21:09.534682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.534651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnsvp" event={"ID":"4367aa03-3c82-49a8-8111-a9f0cd2e124f","Type":"ContainerStarted","Data":"a4f35af4c95bb3c7f1ccdc3f37dc65cb30845bd653d7d647c0624d75996d10ba"} Aug 05 17:21:09.534834 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.534723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vnsvp" event={"ID":"4367aa03-3c82-49a8-8111-a9f0cd2e124f","Type":"ContainerStarted","Data":"9309271714ab3cc885ecfb395e344acfa32334a31aafd0be74778c54d49f230a"} Aug 05 17:21:09.534834 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.534776 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vnsvp" Aug 05 17:21:09.537443 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.537321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"61cbb50e98717b2da73ec6180fcffff1f44d03ba1138487df0ae999ce2bece4a"} Aug 05 17:21:09.537443 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.537398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"c401fc48cf2abe77558815db6ebd509d9effee46900948cc2a1a3a1da6d1347b"} Aug 05 17:21:09.567192 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.567167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.588536 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.588516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:09.710727 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.710612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.733667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.733624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vnsvp" podStartSLOduration=65.928127469 podStartE2EDuration="1m7.73360773s" totalImagesPullingTime="1.805480261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:09.558324724 +0000 UTC m=+102.989470976" watchObservedRunningTime="2026-08-05 17:21:09.73360773 +0000 UTC m=+103.164753979" Aug 05 17:21:09.734004 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.733988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-l9kqb"] Aug 05 17:21:09.737125 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:09.737103 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64bd4b2_fbd2_41c2_beb8_9da5351adb1f.slice/crio-65815fe1de310bb54deb8bdfd399ae83ba641e28e9a4724f06e469bc0ef38969 WatchSource:0}: Error finding container 65815fe1de310bb54deb8bdfd399ae83ba641e28e9a4724f06e469bc0ef38969: Status 404 returned error can't find the container with id 65815fe1de310bb54deb8bdfd399ae83ba641e28e9a4724f06e469bc0ef38969 Aug 05 17:21:09.811750 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:09.811721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.124740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.124652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.224549 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.224462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.423007 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.422915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.545355 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.545297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-l9kqb" event={"ID":"b64bd4b2-fbd2-41c2-beb8-9da5351adb1f","Type":"ContainerStarted","Data":"65815fe1de310bb54deb8bdfd399ae83ba641e28e9a4724f06e469bc0ef38969"} Aug 05 17:21:10.548609 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.548533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" event={"ID":"7de704b1-f978-4db8-9043-9e37630d4aa4","Type":"ContainerStarted","Data":"3351c16fb8baad9a4fd0521d78c8f7bfdc313cf0eb919f843caad0705e98c391"} Aug 05 17:21:10.554110 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.553950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"d9a612a2bf2765f59333b6c99dbfb7486598870568297c660b65a275f31c0f03"} Aug 05 17:21:10.554110 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.554043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"2ba7a598f24d1a2c931ed90ee2acfefffffa1509f162c14d97993c851fc4612f"} Aug 05 17:21:10.554110 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.554074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"ff4edde02eb0eb314598960f9b9924e5cc3d2ad3307136a33a29c29aacce9192"} Aug 05 17:21:10.576808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.576778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.581552 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.581530 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:21:10.585537 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.585306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.588201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.588174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 05 17:21:10.588421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.588385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7to3f137h4h2a\"" Aug 05 17:21:10.588678 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.588660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 05 17:21:10.588749 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.588705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 05 17:21:10.589105 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.588868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 05 17:21:10.589105 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 05 17:21:10.589446 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 05 17:21:10.589530 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589465 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 05 17:21:10.589530 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 05 17:21:10.589530 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589507 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 05 17:21:10.589688 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x7ccb\"" Aug 05 17:21:10.589688 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589663 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 05 17:21:10.589918 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.589876 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 05 17:21:10.593229 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.592130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 05 17:21:10.593323 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.593272 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 05 17:21:10.597829 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.596728 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 05 17:21:10.599730 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.599618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 05 17:21:10.605604 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.605582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:21:10.644444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.644380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:10.666075 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.666284 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.666503 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.666577 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.666776 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.666947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6c45\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.667888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.668847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.672911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.669015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.770828 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.770828 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.770980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c45\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.771417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.771342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.772165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.772142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.774462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.773209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.774462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.774206 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.776320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.775918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.776426 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.776376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.777996 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.777746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.778650 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.778600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.778879 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.778854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.780618 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.779108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.780748 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.779691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.781843 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.781812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.782010 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.781987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.782218 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.782195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.782695 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.782674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.783169 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.783128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.783480 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.783421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.783480 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.783433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.783732 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.783690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.785838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.785818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.795522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.795494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c45\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45\") pod \"prometheus-k8s-0\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.901444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.901406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:10.934611 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:10.934472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.140046 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.139980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:21:11.144563 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:11.144524 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a66bbf_e159_4a83_90f1_a5d9cbc0a68d.slice/crio-d858879f44d0f35b2efedce922a606e9c595616c2c75b7fd3fd7fc4c6646270d WatchSource:0}: Error finding container d858879f44d0f35b2efedce922a606e9c595616c2c75b7fd3fd7fc4c6646270d: Status 404 returned error can't find the container with id d858879f44d0f35b2efedce922a606e9c595616c2c75b7fd3fd7fc4c6646270d Aug 05 17:21:11.172520 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.172411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.265887 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.265854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.439831 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.439799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.453855 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.453828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dd84f8cf4-r96wh" Aug 05 17:21:11.562836 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.562153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerStarted","Data":"e5f8f4605d70b98f3cd9f93a2730c5a0af6245e37a6feaa214693c2b696987b5"} Aug 05 17:21:11.563692 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.563670 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" exitCode=0 Aug 05 17:21:11.563800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.563732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} Aug 05 17:21:11.563800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.563792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"d858879f44d0f35b2efedce922a606e9c595616c2c75b7fd3fd7fc4c6646270d"} Aug 05 17:21:11.565330 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.565305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" event={"ID":"7de704b1-f978-4db8-9043-9e37630d4aa4","Type":"ContainerStarted","Data":"2b742f7dca3d799e9f2e8e51f2c57683e9bfb2a6641987b24b18d7c76b9629f1"} Aug 05 17:21:11.565585 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.565541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:11.570718 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.570698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" Aug 05 17:21:11.602016 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.601972 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.553237384 podStartE2EDuration="6.601940598s" totalImagesPullingTime="4.030821276s" totalInitContainerRuntime="17.881938ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:11.600858194 +0000 UTC m=+105.032004471" watchObservedRunningTime="2026-08-05 17:21:11.601940598 +0000 UTC m=+105.033086908" Aug 05 17:21:11.625054 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.624796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.725381 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:11.725301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:12.654491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:12.654458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:12.713890 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:12.713859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:14.210473 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:14.210403 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" podUID="fb294dab-5273-492e-b688-524ea8c3f6d7" containerName="registry" containerID="cri-o://255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce" gracePeriod=30 Aug 05 17:21:15.295815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.295738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.299787 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.299764 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:21:15.325222 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.324318 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-s87fv" podStartSLOduration=4.922442512 podStartE2EDuration="6.324297048s" totalImagesPullingTime="1.401854536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:11.67422016 +0000 UTC m=+105.105366411" watchObservedRunningTime="2026-08-05 17:21:15.324297048 +0000 UTC m=+108.755443302" Aug 05 17:21:15.398429 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.398332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.417252 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwkvm\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417320 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417353 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417401 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417593 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417593 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.417593 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.417519 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets\") pod \"fb294dab-5273-492e-b688-524ea8c3f6d7\" (UID: \"fb294dab-5273-492e-b688-524ea8c3f6d7\") " Aug 05 17:21:15.418931 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.418811 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:21:15.418931 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.418820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:21:15.421526 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.421481 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:21:15.421728 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.421669 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:21:15.421939 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.421899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:15.422090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.422060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:15.422232 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.422195 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm" (OuterVolumeSpecName: "kube-api-access-nwkvm") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "kube-api-access-nwkvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:15.431296 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.431266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fb294dab-5273-492e-b688-524ea8c3f6d7" (UID: "fb294dab-5273-492e-b688-524ea8c3f6d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:15.519093 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519055 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-image-registry-private-configuration\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519093 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519091 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkvm\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-kube-api-access-nwkvm\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519107 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-trusted-ca\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519116 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-certificates\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519124 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb294dab-5273-492e-b688-524ea8c3f6d7-ca-trust-extracted\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519132 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-bound-sa-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519139 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb294dab-5273-492e-b688-524ea8c3f6d7-registry-tls\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.519328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.519147 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb294dab-5273-492e-b688-524ea8c3f6d7-installation-pull-secrets\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:21:15.593445 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.593407 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb294dab-5273-492e-b688-524ea8c3f6d7" containerID="255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce" exitCode=0 Aug 05 17:21:15.593616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.593534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" event={"ID":"fb294dab-5273-492e-b688-524ea8c3f6d7","Type":"ContainerDied","Data":"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce"} Aug 05 17:21:15.593616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.593587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" event={"ID":"fb294dab-5273-492e-b688-524ea8c3f6d7","Type":"ContainerDied","Data":"0bc85b36f1374fe07ce1b2b80013e6a0ede8b902febd9b73cc23be05a0b58323"} Aug 05 17:21:15.593616 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.593613 2564 scope.go:117] "RemoveContainer" containerID="255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce" Aug 05 17:21:15.593779 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.593763 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9df757c4-96h9g" Aug 05 17:21:15.607568 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.607138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} Aug 05 17:21:15.607568 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.607219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} Aug 05 17:21:15.613523 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.613497 2564 scope.go:117] "RemoveContainer" containerID="255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce" Aug 05 17:21:15.614091 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:21:15.613872 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce\": container with ID starting with 255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce not found: ID does not exist" containerID="255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce" Aug 05 17:21:15.614091 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.613914 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce"} err="failed to get container status \"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce\": rpc error: code = NotFound desc = could not find container \"255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce\": container with ID starting with 255996cc4264cabbafed5f78c1524b70580f418719ab7064cc79060caaab18ce not found: ID does not exist" Aug 05 17:21:15.632406 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.632381 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:21:15.642336 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:15.642243 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f9df757c4-96h9g"] Aug 05 17:21:16.752104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:16.752072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:17.039549 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.039471 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb294dab-5273-492e-b688-524ea8c3f6d7" path="/var/lib/kubelet/pods/fb294dab-5273-492e-b688-524ea8c3f6d7/volumes" Aug 05 17:21:17.624873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.624837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} Aug 05 17:21:17.625053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.624920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} Aug 05 17:21:17.625053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.624981 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} Aug 05 17:21:17.625053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.625022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerStarted","Data":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} Aug 05 17:21:17.660435 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:17.659505 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.263261922 podStartE2EDuration="7.659485777s" totalImagesPullingTime="5.370594498s" totalInitContainerRuntime="25.629357ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:17.657851084 +0000 UTC m=+111.088997333" watchObservedRunningTime="2026-08-05 17:21:17.659485777 +0000 UTC m=+111.090632031" Aug 05 17:21:19.557110 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:19.557033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vnsvp" Aug 05 17:21:20.901723 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:20.901687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:21:24.766683 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:24.766646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:24.891113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:24.891073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:26.222090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:26.222054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:26.321606 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:26.321567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:27.673509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:27.673468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-l9kqb" event={"ID":"b64bd4b2-fbd2-41c2-beb8-9da5351adb1f","Type":"ContainerStarted","Data":"ef11c09080f4c5891fc4e4d74fdf4b99270fde97a6fd2b1c16445f86cdcbc6b5"} Aug 05 17:21:27.674003 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:27.673572 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:27.685744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:27.685711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-l9kqb" Aug 05 17:21:27.711862 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:27.711805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-l9kqb" podStartSLOduration=1.776984538 podStartE2EDuration="18.711788633s" totalImagesPullingTime="16.934804095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:27.692890205 +0000 UTC m=+121.124036455" watchObservedRunningTime="2026-08-05 17:21:27.711788633 +0000 UTC m=+121.142934882" Aug 05 17:21:30.287794 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:30.287760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:30.388710 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:30.388674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:32.150159 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:32.150122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:32.228167 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:32.228133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:32.940250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:32.940213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.040548 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:33.040491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.743338 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:33.743297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.848789 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:33.848757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.946040 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:33.945996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:34.563109 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:34.563073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:34.660889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:34.660856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:35.152464 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:35.152431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:36.916829 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:36.916797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:21:36.919113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:36.919096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3311739-a779-4c3d-a848-5482e3cee472-metrics-certs\") pod \"network-metrics-daemon-2hc9s\" (UID: \"a3311739-a779-4c3d-a848-5482e3cee472\") " pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:21:36.948914 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:36.948889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rdxm\"" Aug 05 17:21:36.956618 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:36.956601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hc9s" Aug 05 17:21:37.116129 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:37.116100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hc9s"] Aug 05 17:21:37.118117 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:21:37.118085 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3311739_a779_4c3d_a848_5482e3cee472.slice/crio-9a5aa6268b7cae4514bc41b3742e37c40ee39c17ec25db3940777ecd772da01a WatchSource:0}: Error finding container 9a5aa6268b7cae4514bc41b3742e37c40ee39c17ec25db3940777ecd772da01a: Status 404 returned error can't find the container with id 9a5aa6268b7cae4514bc41b3742e37c40ee39c17ec25db3940777ecd772da01a Aug 05 17:21:37.713853 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:37.713819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hc9s" event={"ID":"a3311739-a779-4c3d-a848-5482e3cee472","Type":"ContainerStarted","Data":"9a5aa6268b7cae4514bc41b3742e37c40ee39c17ec25db3940777ecd772da01a"} Aug 05 17:21:39.724188 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:39.724143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hc9s" event={"ID":"a3311739-a779-4c3d-a848-5482e3cee472","Type":"ContainerStarted","Data":"9d7f240c24a216d2d0ee5b40d8d87ea56d1824715a4c03f0b4e4c25e59d825b9"} Aug 05 17:21:39.724771 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:39.724217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hc9s" event={"ID":"a3311739-a779-4c3d-a848-5482e3cee472","Type":"ContainerStarted","Data":"59542503268ead61ac543eeba75ecb3f287dd04479548662b96d1701128344f4"} Aug 05 17:21:39.743774 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:39.743721 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2hc9s" podStartSLOduration=131.255509345 podStartE2EDuration="2m12.743704439s" totalImagesPullingTime="1.488195094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:39.742293223 +0000 UTC m=+133.173439475" watchObservedRunningTime="2026-08-05 17:21:39.743704439 +0000 UTC m=+133.174850688" Aug 05 17:21:39.854919 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:39.854883 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:39.957372 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:39.957331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:40.054885 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:40.054855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:40.728412 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:40.728381 2564 generic.go:356] "Generic (PLEG): container finished" podID="128344e9-2717-46e4-a3eb-6abcc088efc0" containerID="2ba02baed86c02bfc3f0b18c94e9aa2f5d6c527b016cac229fb569845c0407c8" exitCode=0 Aug 05 17:21:40.728756 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:40.728454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" event={"ID":"128344e9-2717-46e4-a3eb-6abcc088efc0","Type":"ContainerDied","Data":"2ba02baed86c02bfc3f0b18c94e9aa2f5d6c527b016cac229fb569845c0407c8"} Aug 05 17:21:40.728878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:40.728864 2564 scope.go:117] "RemoveContainer" containerID="2ba02baed86c02bfc3f0b18c94e9aa2f5d6c527b016cac229fb569845c0407c8" Aug 05 17:21:41.734908 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:41.734878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-vtcnn" event={"ID":"128344e9-2717-46e4-a3eb-6abcc088efc0","Type":"ContainerStarted","Data":"efe91abf0b1cf2e4a42d77e1f888bbff196acc1d0af01c530964dc695ca01650"} Aug 05 17:21:42.792481 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:42.792447 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d8d8f5b-rrnmh_6462a8f5-6939-4711-9294-168ea9216841/router/0.log" Aug 05 17:21:42.799777 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:42.799752 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6xz5_8927124e-9aee-4387-b5f1-cbcd8b6782b4/serve-healthcheck-canary/0.log" Aug 05 17:21:45.295287 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:45.295252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:45.360906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:45.360878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:54.226451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:54.226410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:54.326237 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:54.326205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:54.401673 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:21:54.401642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:10.902358 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:10.902324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:10.917825 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:10.917804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:11.857748 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:11.857724 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:20.414691 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:20.414615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:20.515466 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:20.515434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:24.721230 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721202 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:22:24.721681 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721560 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="alertmanager" containerID="cri-o://c401fc48cf2abe77558815db6ebd509d9effee46900948cc2a1a3a1da6d1347b" gracePeriod=120 Aug 05 17:22:24.721681 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721620 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-metric" containerID="cri-o://d9a612a2bf2765f59333b6c99dbfb7486598870568297c660b65a275f31c0f03" gracePeriod=120 Aug 05 17:22:24.721681 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721620 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-web" containerID="cri-o://ff4edde02eb0eb314598960f9b9924e5cc3d2ad3307136a33a29c29aacce9192" gracePeriod=120 Aug 05 17:22:24.721681 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721644 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="config-reloader" containerID="cri-o://61cbb50e98717b2da73ec6180fcffff1f44d03ba1138487df0ae999ce2bece4a" gracePeriod=120 Aug 05 17:22:24.721841 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721661 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy" containerID="cri-o://2ba7a598f24d1a2c931ed90ee2acfefffffa1509f162c14d97993c851fc4612f" gracePeriod=120 Aug 05 17:22:24.721896 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.721838 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="prom-label-proxy" containerID="cri-o://e5f8f4605d70b98f3cd9f93a2730c5a0af6245e37a6feaa214693c2b696987b5" gracePeriod=120 Aug 05 17:22:24.892299 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892267 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="e5f8f4605d70b98f3cd9f93a2730c5a0af6245e37a6feaa214693c2b696987b5" exitCode=0 Aug 05 17:22:24.892299 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892295 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="2ba7a598f24d1a2c931ed90ee2acfefffffa1509f162c14d97993c851fc4612f" exitCode=0 Aug 05 17:22:24.892299 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892303 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="61cbb50e98717b2da73ec6180fcffff1f44d03ba1138487df0ae999ce2bece4a" exitCode=0 Aug 05 17:22:24.892447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892308 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="c401fc48cf2abe77558815db6ebd509d9effee46900948cc2a1a3a1da6d1347b" exitCode=0 Aug 05 17:22:24.892447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"e5f8f4605d70b98f3cd9f93a2730c5a0af6245e37a6feaa214693c2b696987b5"} Aug 05 17:22:24.892447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"2ba7a598f24d1a2c931ed90ee2acfefffffa1509f162c14d97993c851fc4612f"} Aug 05 17:22:24.892447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"61cbb50e98717b2da73ec6180fcffff1f44d03ba1138487df0ae999ce2bece4a"} Aug 05 17:22:24.892447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:24.892445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"c401fc48cf2abe77558815db6ebd509d9effee46900948cc2a1a3a1da6d1347b"} Aug 05 17:22:25.900485 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:25.900460 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="d9a612a2bf2765f59333b6c99dbfb7486598870568297c660b65a275f31c0f03" exitCode=0 Aug 05 17:22:25.900485 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:25.900481 2564 generic.go:356] "Generic (PLEG): container finished" podID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerID="ff4edde02eb0eb314598960f9b9924e5cc3d2ad3307136a33a29c29aacce9192" exitCode=0 Aug 05 17:22:25.900944 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:25.900532 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"d9a612a2bf2765f59333b6c99dbfb7486598870568297c660b65a275f31c0f03"} Aug 05 17:22:25.900944 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:25.900628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"ff4edde02eb0eb314598960f9b9924e5cc3d2ad3307136a33a29c29aacce9192"} Aug 05 17:22:25.959322 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:25.959301 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:22:26.071911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.071911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.071911 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071904 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071974 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.071999 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072034 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vzl\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072122 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072793 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072224 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca\") pod \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\" (UID: \"288ebc5a-20d7-4c0d-996e-3573ebcf06f5\") " Aug 05 17:22:26.072793 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072328 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:26.072793 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072743 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.072949 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.072826 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:26.073375 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.073345 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:26.075842 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.075815 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.075937 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.075890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:26.076024 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.075947 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.076024 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.076005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.076262 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.076231 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl" (OuterVolumeSpecName: "kube-api-access-m8vzl") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "kube-api-access-m8vzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:26.076262 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.076253 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume" (OuterVolumeSpecName: "config-volume") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.076414 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.076369 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out" (OuterVolumeSpecName: "config-out") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:26.076482 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.076458 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.079941 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.079921 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.085518 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.085497 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config" (OuterVolumeSpecName: "web-config") pod "288ebc5a-20d7-4c0d-996e-3573ebcf06f5" (UID: "288ebc5a-20d7-4c0d-996e-3573ebcf06f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:26.174001 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.173980 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-web-config\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174001 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.173996 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-out\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174007 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174016 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-alertmanager-main-db\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174025 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vzl\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-kube-api-access-m8vzl\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174033 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-tls-assets\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174042 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174049 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-cluster-tls-config\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174057 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-metrics-client-ca\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174064 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-main-tls\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174072 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-config-volume\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.174107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.174080 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/288ebc5a-20d7-4c0d-996e-3573ebcf06f5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:26.910169 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.910139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"288ebc5a-20d7-4c0d-996e-3573ebcf06f5","Type":"ContainerDied","Data":"d9a9e81a96feac130498fc4be2c1c31a9a37a26c9ed309febda20296cc81c6cd"} Aug 05 17:22:26.910586 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.910225 2564 scope.go:117] "RemoveContainer" containerID="e5f8f4605d70b98f3cd9f93a2730c5a0af6245e37a6feaa214693c2b696987b5" Aug 05 17:22:26.910586 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.910238 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 05 17:22:26.919073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.918683 2564 scope.go:117] "RemoveContainer" containerID="d9a612a2bf2765f59333b6c99dbfb7486598870568297c660b65a275f31c0f03" Aug 05 17:22:26.930219 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.930203 2564 scope.go:117] "RemoveContainer" containerID="2ba7a598f24d1a2c931ed90ee2acfefffffa1509f162c14d97993c851fc4612f" Aug 05 17:22:26.936600 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.936583 2564 scope.go:117] "RemoveContainer" containerID="ff4edde02eb0eb314598960f9b9924e5cc3d2ad3307136a33a29c29aacce9192" Aug 05 17:22:26.943098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.943080 2564 scope.go:117] "RemoveContainer" containerID="61cbb50e98717b2da73ec6180fcffff1f44d03ba1138487df0ae999ce2bece4a" Aug 05 17:22:26.949418 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.949398 2564 scope.go:117] "RemoveContainer" containerID="c401fc48cf2abe77558815db6ebd509d9effee46900948cc2a1a3a1da6d1347b" Aug 05 17:22:26.953841 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.953824 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:22:26.956479 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.956466 2564 scope.go:117] "RemoveContainer" containerID="b3791d7a000fcd751a12bb1a138e38f6a852d305d26ead4611fcc9782e77fbfe" Aug 05 17:22:26.972096 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.972078 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 05 17:22:26.986630 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:26.986610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:27.038802 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:27.038779 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" path="/var/lib/kubelet/pods/288ebc5a-20d7-4c0d-996e-3573ebcf06f5/volumes" Aug 05 17:22:27.163315 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:27.163232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:28.964890 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.964860 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:28.965385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965239 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="prometheus" containerID="cri-o://613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" gracePeriod=600 Aug 05 17:22:28.965385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965255 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy" containerID="cri-o://3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" gracePeriod=600 Aug 05 17:22:28.965385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965267 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="thanos-sidecar" containerID="cri-o://86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" gracePeriod=600 Aug 05 17:22:28.965385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965295 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="config-reloader" containerID="cri-o://68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" gracePeriod=600 Aug 05 17:22:28.965385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965270 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" gracePeriod=600 Aug 05 17:22:28.965625 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:28.965401 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-web" containerID="cri-o://42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" gracePeriod=600 Aug 05 17:22:29.201779 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.201755 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:29.298547 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298606 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298632 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.298697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298681 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6c45\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298783 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298821 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.298996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299715 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299093 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299715 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls\") pod \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\" (UID: \"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d\") " Aug 05 17:22:29.299828 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299750 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.299887 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.299867 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.300385 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.300356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.300614 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.300592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.300985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301276 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301301 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-metrics-client-ca\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301320 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301333 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301349 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.301390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301362 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.301865 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301442 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:29.301865 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301695 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:29.302035 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.301994 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.302376 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.302344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out" (OuterVolumeSpecName: "config-out") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:29.302830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.302759 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45" (OuterVolumeSpecName: "kube-api-access-s6c45") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "kube-api-access-s6c45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:29.302830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.302785 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config" (OuterVolumeSpecName: "config") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.303133 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.303110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.303570 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.303549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.303667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.303569 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:29.303998 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.303972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.304200 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.304166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.304446 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.304429 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.304630 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.304606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.312876 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.312855 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config" (OuterVolumeSpecName: "web-config") pod "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" (UID: "b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:29.401834 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401804 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.401834 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401832 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401845 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-tls-assets\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401854 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-web-config\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401864 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401872 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-grpc-tls\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401880 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config-out\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401888 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-metrics-client-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401896 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401905 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401920 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-kube-rbac-proxy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401930 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401938 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-config\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401945 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6c45\" (UniqueName: \"kubernetes.io/projected/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-kube-api-access-s6c45\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.402008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.401971 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d-prometheus-k8s-db\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:22:29.924292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924260 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" exitCode=0 Aug 05 17:22:29.924292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924286 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" exitCode=0 Aug 05 17:22:29.924292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924294 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" exitCode=0 Aug 05 17:22:29.924292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924300 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" exitCode=0 Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924305 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" exitCode=0 Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924310 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" exitCode=0 Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924327 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924481 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} Aug 05 17:22:29.924555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} Aug 05 17:22:29.924930 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.924560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d","Type":"ContainerDied","Data":"d858879f44d0f35b2efedce922a606e9c595616c2c75b7fd3fd7fc4c6646270d"} Aug 05 17:22:29.932499 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.932480 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.939292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.939276 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.945782 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.945764 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.948134 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.948116 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:29.952849 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.952836 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.953548 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.953532 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:29.960807 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.960202 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.967766 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.967590 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.974033 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974017 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.974288 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.974271 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.974341 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974296 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.974341 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974315 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.974537 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.974520 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.974594 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974540 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.974594 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974557 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.974774 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.974758 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.974839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.974839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.974788 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.975075 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.975058 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.975161 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975102 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.975161 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975123 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.975368 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.975352 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.975422 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975370 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.975422 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975386 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.975577 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.975565 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.975621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975579 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.975621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975591 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.975797 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:22:29.975782 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.975839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975799 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.975839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.975811 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.976035 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976016 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.976082 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976037 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.976235 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976218 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.976275 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976237 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.976430 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976415 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.976479 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976430 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.976638 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976618 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.976675 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976639 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.976805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976785 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.976889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.976807 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.977053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977035 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.977102 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977055 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.977229 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977211 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.977292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977230 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.977438 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977399 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.977438 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977417 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.977644 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977628 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.977690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.977644 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.978073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978054 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.978161 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978074 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.978330 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978302 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.978330 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978320 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.978489 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978471 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.978550 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978491 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.978715 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978696 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.978793 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978716 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.978993 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978976 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.979054 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.978995 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.979232 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979212 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.979289 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979233 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.979477 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979461 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.979538 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979480 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.979739 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979723 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.979812 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979741 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.979988 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979972 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.980055 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.979990 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.980225 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980206 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.980297 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980227 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.980477 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980459 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.980529 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980478 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.980726 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980710 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.980788 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980728 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.981003 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.980942 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.981067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.981007 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.981269 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.981241 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.981269 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.981259 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.981638 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.981594 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.981638 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.981620 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.982527 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.982506 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.982621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.982530 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.982778 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.982757 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.982848 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.982779 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.983075 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983057 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.983151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983076 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.983357 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983332 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.983458 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983358 2564 scope.go:117] "RemoveContainer" containerID="9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7" Aug 05 17:22:29.983622 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983603 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7"} err="failed to get container status \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": rpc error: code = NotFound desc = could not find container \"9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7\": container with ID starting with 9f518ece86894e0e26e200856540b827185924688ab541fc6399cb27f7b14cf7 not found: ID does not exist" Aug 05 17:22:29.983672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983623 2564 scope.go:117] "RemoveContainer" containerID="3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207" Aug 05 17:22:29.984080 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983920 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207"} err="failed to get container status \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": rpc error: code = NotFound desc = could not find container \"3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207\": container with ID starting with 3f501a43c547c612654704b911541d8786ec75cf9be4a18c64b19cd9cc673207 not found: ID does not exist" Aug 05 17:22:29.984080 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.983946 2564 scope.go:117] "RemoveContainer" containerID="42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261" Aug 05 17:22:29.984228 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984138 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:29.984319 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984235 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261"} err="failed to get container status \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": rpc error: code = NotFound desc = could not find container \"42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261\": container with ID starting with 42986f820a448fe8adb31782191c51cdcd9ce059e9db864c818518b80b770261 not found: ID does not exist" Aug 05 17:22:29.984319 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984257 2564 scope.go:117] "RemoveContainer" containerID="86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984452 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984468 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984482 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="prometheus" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984491 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="prometheus" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984504 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984513 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984529 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb294dab-5273-492e-b688-524ea8c3f6d7" containerName="registry" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984536 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb294dab-5273-492e-b688-524ea8c3f6d7" containerName="registry" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984542 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="init-config-reloader" Aug 05 17:22:29.984555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984555 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="init-config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984562 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="alertmanager" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984567 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="alertmanager" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984574 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984582 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984504 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96"} err="failed to get container status \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": rpc error: code = NotFound desc = could not find container \"86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96\": container with ID starting with 86f921e19000a540db4ee4f4f8e7bea9e08864b921cc0676041ff5e5de840f96 not found: ID does not exist" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984592 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984597 2564 scope.go:117] "RemoveContainer" containerID="68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984600 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984648 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="thanos-sidecar" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984654 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="thanos-sidecar" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984662 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="init-config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984670 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="init-config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984681 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984689 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984697 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984705 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984718 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-metric" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984725 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-metric" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984732 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="prom-label-proxy" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984739 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="prom-label-proxy" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984749 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-thanos" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984756 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-thanos" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984825 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="prom-label-proxy" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984834 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984842 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984849 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="config-reloader" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984857 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="thanos-sidecar" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984864 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="alertmanager" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984870 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="prometheus" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984877 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb294dab-5273-492e-b688-524ea8c3f6d7" containerName="registry" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984884 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy-metric" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984891 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-web" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984898 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" containerName="kube-rbac-proxy-thanos" Aug 05 17:22:29.985363 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984928 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="kube-rbac-proxy" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.984936 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288ebc5a-20d7-4c0d-996e-3573ebcf06f5" containerName="config-reloader" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.985090 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4"} err="failed to get container status \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": rpc error: code = NotFound desc = could not find container \"68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4\": container with ID starting with 68cbf1b96e227eb85d3d1e06917a95bf3b50a8d4e47737af5079af382f485ae4 not found: ID does not exist" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.985113 2564 scope.go:117] "RemoveContainer" containerID="613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.985399 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7"} err="failed to get container status \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": rpc error: code = NotFound desc = could not find container \"613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7\": container with ID starting with 613512fa917ba5608587de7aba9d754b10f4302166ff92adcc0e544116497be7 not found: ID does not exist" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.985424 2564 scope.go:117] "RemoveContainer" containerID="257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61" Aug 05 17:22:29.986933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.986006 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61"} err="failed to get container status \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": rpc error: code = NotFound desc = could not find container \"257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61\": container with ID starting with 257bcc1d9f0590f501ec2047fcc9b21bb3fc69867c752acb9755a9231d621e61 not found: ID does not exist" Aug 05 17:22:29.991564 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.991545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:29.996422 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.996406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 05 17:22:29.997151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 05 17:22:29.997151 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997070 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 05 17:22:29.997285 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 05 17:22:29.997285 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997271 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 05 17:22:29.997562 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997543 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 05 17:22:29.997653 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 05 17:22:29.997653 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.997573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 05 17:22:29.998398 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.998380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7to3f137h4h2a\"" Aug 05 17:22:29.998474 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.998405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 05 17:22:29.998474 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.998405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 05 17:22:29.998579 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:29.998406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 05 17:22:30.007101 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.006928 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 05 17:22:30.007101 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.006942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 05 17:22:30.007243 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.007063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x7ccb\"" Aug 05 17:22:30.007243 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.007157 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 05 17:22:30.010995 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.010978 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 05 17:22:30.031266 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.031243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:30.108390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.108943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.109264 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.108994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj66m\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-kube-api-access-gj66m\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209687 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209687 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209687 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209687 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209938 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209938 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.209938 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.209735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj66m\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-kube-api-access-gj66m\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210537 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.210804 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.210702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.211154 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.211130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.212989 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.212670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.212989 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.212833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.213171 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.213036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.213832 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.213801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.214185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.214573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.214680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.214944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215370 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215710 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215747 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-config\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215791 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.215825 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.215808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.216104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.216090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.219995 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.219977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj66m\" (UniqueName: \"kubernetes.io/projected/a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41-kube-api-access-gj66m\") pod \"prometheus-k8s-0\" (UID: \"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.309161 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.309137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:30.442694 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.442665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 05 17:22:30.445545 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:22:30.445517 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f9d50b_d5fb_44cc_a977_c8bf05a0fe41.slice/crio-4506afa31576a12d80fae91b3f4a7fcc144fe092eac50957cfe1f22163fe4095 WatchSource:0}: Error finding container 4506afa31576a12d80fae91b3f4a7fcc144fe092eac50957cfe1f22163fe4095: Status 404 returned error can't find the container with id 4506afa31576a12d80fae91b3f4a7fcc144fe092eac50957cfe1f22163fe4095 Aug 05 17:22:30.928926 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.928889 2564 generic.go:356] "Generic (PLEG): container finished" podID="a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41" containerID="6f19c6352d2c418e6fe0c600d959d3ff1ab18a1046d817d3ee94dfd1726866b3" exitCode=0 Aug 05 17:22:30.929124 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.929042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerDied","Data":"6f19c6352d2c418e6fe0c600d959d3ff1ab18a1046d817d3ee94dfd1726866b3"} Aug 05 17:22:30.929124 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:30.929115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"4506afa31576a12d80fae91b3f4a7fcc144fe092eac50957cfe1f22163fe4095"} Aug 05 17:22:31.038132 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.038107 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d" path="/var/lib/kubelet/pods/b7a66bbf-e159-4a83-90f1-a5d9cbc0a68d/volumes" Aug 05 17:22:31.935951 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.935925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"47168d427b1d8c924ef98bfd19a301a9c3019ff564e740dbc6755fec9cbab832"} Aug 05 17:22:31.936128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.936006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"44ccf8b889e9c9124dba16ca482453657636a8c0ba33f8f77189f345923d5c86"} Aug 05 17:22:31.936128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.936032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"385e531752a74dd4c90d85eb6a8b5b04691094a27bab64eb4b15fbd44021e190"} Aug 05 17:22:31.936128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.936055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"83e9e087310dc418d31c9be458666810b758167b5bbbd99cccd201f98bca8b9c"} Aug 05 17:22:31.936128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.936082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"2f65928569371b3e15469c9d87bb5bbda5e5a838eb21e5ee3301a89e56da03a2"} Aug 05 17:22:31.936128 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.936104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41","Type":"ContainerStarted","Data":"f908fddf899f226efe67740ba3f7f986fc0226496eeb07db7a55ff203e4a841a"} Aug 05 17:22:31.970468 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:31.970419 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.949509978 podStartE2EDuration="2.970404438s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.89446ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:31.969157286 +0000 UTC m=+185.400303555" watchObservedRunningTime="2026-08-05 17:22:31.970404438 +0000 UTC m=+185.401550704" Aug 05 17:22:32.444559 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:32.444516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:32.574711 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:32.574675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:35.310233 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:35.310207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:22:38.156760 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:38.156730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:38.259985 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:38.259935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:38.354060 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:38.354032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:45.297451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:45.297417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:45.408894 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:45.408859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:54.339057 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:22:54.339020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:12.800055 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:12.800014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:12.901396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:12.901375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:23:30.309886 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:30.309840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:23:30.325261 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:30.325236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:23:31.149575 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:31.149550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 05 17:23:55.805983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:23:55.805887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:20.168337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:20.168302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:20.270189 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:20.270159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:26.919685 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:26.919648 2564 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 17:24:42.155907 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:42.155872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:42.930094 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:42.930064 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:45.736159 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:45.736126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:45.843568 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:45.843533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:46.638158 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:46.638122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:46.736057 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:46.736018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:47.504278 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:47.504244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:48.556169 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:48.556129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:49.560436 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:49.560390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:49.911093 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:49.911013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:52.722881 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:52.722852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:52.822735 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:52.822706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:58.122315 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:58.122281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:58.410468 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:58.410389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:58.511527 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:58.511498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:59.141228 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:59.141196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:24:59.224440 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:24:59.224404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:04.286199 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:04.286165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:04.387277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:04.387253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:06.597173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:06.597141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:22.134672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:22.134585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:22.236719 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:22.236689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:22.737699 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:22.737668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.033254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.033175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.133405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.133379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.143086 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.143054 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx"] Aug 05 17:25:23.146057 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.146040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.148855 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.148839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 17:25:23.148946 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.148923 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 17:25:23.149462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.149441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9vhvx\"" Aug 05 17:25:23.149680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.149664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 17:25:23.149747 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.149694 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 17:25:23.163298 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.163278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx"] Aug 05 17:25:23.251089 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.251064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.251201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.251106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.251201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.251167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbfv\" (UniqueName: \"kubernetes.io/projected/b41391f9-a45c-4bb7-9c3f-d61787772f17-kube-api-access-2gbfv\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.251201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.251187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-tmp\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.251303 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.251216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-manifests\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.282083 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.282060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.351758 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.351737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.351838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.351767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.351838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.351797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbfv\" (UniqueName: \"kubernetes.io/projected/b41391f9-a45c-4bb7-9c3f-d61787772f17-kube-api-access-2gbfv\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.351838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.351814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-tmp\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.351838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.351834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-manifests\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.352165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.352148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-manifests\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.354216 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.354197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41391f9-a45c-4bb7-9c3f-d61787772f17-tmp\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.354380 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.354366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.354490 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.354472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b41391f9-a45c-4bb7-9c3f-d61787772f17-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.364158 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.364135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbfv\" (UniqueName: \"kubernetes.io/projected/b41391f9-a45c-4bb7-9c3f-d61787772f17-kube-api-access-2gbfv\") pod \"opendatahub-operator-controller-manager-58c579b484-94lvx\" (UID: \"b41391f9-a45c-4bb7-9c3f-d61787772f17\") " pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.384407 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.384385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.456164 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.456144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:23.482692 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.482660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.583990 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.583932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.584081 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.583998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx"] Aug 05 17:25:23.586608 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:23.586586 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41391f9_a45c_4bb7_9c3f_d61787772f17.slice/crio-92ce21cf629d6158e0de9100523e28a7af436f6a121cac24feba6fbc4f52ff42 WatchSource:0}: Error finding container 92ce21cf629d6158e0de9100523e28a7af436f6a121cac24feba6fbc4f52ff42: Status 404 returned error can't find the container with id 92ce21cf629d6158e0de9100523e28a7af436f6a121cac24feba6fbc4f52ff42 Aug 05 17:25:23.588342 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.588327 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:25:23.695586 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.695523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.787400 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.787379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:23.885498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:23.885475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:24.519035 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:24.519000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" event={"ID":"b41391f9-a45c-4bb7-9c3f-d61787772f17","Type":"ContainerStarted","Data":"92ce21cf629d6158e0de9100523e28a7af436f6a121cac24feba6fbc4f52ff42"} Aug 05 17:25:27.533621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:27.533587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" event={"ID":"b41391f9-a45c-4bb7-9c3f-d61787772f17","Type":"ContainerStarted","Data":"fb2010794d1df90b575d5df35c37e55b2e7a50d82ac197778ca9d313879233cf"} Aug 05 17:25:27.783115 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:27.783083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:28.538412 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:28.538379 2564 generic.go:356] "Generic (PLEG): container finished" podID="b41391f9-a45c-4bb7-9c3f-d61787772f17" containerID="fb2010794d1df90b575d5df35c37e55b2e7a50d82ac197778ca9d313879233cf" exitCode=0 Aug 05 17:25:28.538762 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:28.538447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" event={"ID":"b41391f9-a45c-4bb7-9c3f-d61787772f17","Type":"ContainerDied","Data":"fb2010794d1df90b575d5df35c37e55b2e7a50d82ac197778ca9d313879233cf"} Aug 05 17:25:29.543623 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:29.543586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" event={"ID":"b41391f9-a45c-4bb7-9c3f-d61787772f17","Type":"ContainerStarted","Data":"f0f03000aa2039460ef7a7087810461f1b972da894d142fa089e4f1016458e65"} Aug 05 17:25:29.544081 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:29.543669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:29.567153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:29.567111 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" podStartSLOduration=2.047045068 podStartE2EDuration="6.567096749s" totalImagesPullingTime="3.595870817s" totalInitContainerRuntime="924.180864ms" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:29.564679991 +0000 UTC m=+362.995826240" watchObservedRunningTime="2026-08-05 17:25:29.567096749 +0000 UTC m=+362.998243000" Aug 05 17:25:35.138230 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:35.138183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:35.238237 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:35.238196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:40.549615 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:40.549587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58c579b484-94lvx" Aug 05 17:25:42.567836 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:42.567806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:42.721589 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:42.721553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:42.997307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:42.997229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:43.144543 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.144512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:43.556506 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.556474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:43.664106 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.664076 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pkcpp"] Aug 05 17:25:43.672719 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.672703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.694040 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.694022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 05 17:25:43.697547 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.697522 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pkcpp"] Aug 05 17:25:43.707367 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.707349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-llst7\"" Aug 05 17:25:43.780923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.780899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:43.804109 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.804090 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf"] Aug 05 17:25:43.806152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.806138 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:43.811345 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.811303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 17:25:43.812534 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.812515 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 17:25:43.814431 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.814414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 17:25:43.815839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.815824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.815935 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.815919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6js\" (UniqueName: \"kubernetes.io/projected/d7999f09-dd22-470c-b935-36f29873537b-kube-api-access-wg6js\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.816681 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.816664 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 17:25:43.816777 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.816701 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:25:43.816777 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.816772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4hnvq\"" Aug 05 17:25:43.831152 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.831129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf"] Aug 05 17:25:43.916975 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.916927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:43.917088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.916986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.917088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.917014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-metrics-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:43.917088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.917072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-manager-config\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:43.917320 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:25:43.917109 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 05 17:25:43.917320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.917113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzb2g\" (UniqueName: \"kubernetes.io/projected/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-kube-api-access-mzb2g\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:43.917320 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:25:43.917174 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert podName:d7999f09-dd22-470c-b935-36f29873537b nodeName:}" failed. No retries permitted until 2026-08-05 17:25:44.417153991 +0000 UTC m=+377.848300220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert") pod "odh-model-controller-7998c67dc4-pkcpp" (UID: "d7999f09-dd22-470c-b935-36f29873537b") : secret "odh-model-controller-webhook-cert" not found Aug 05 17:25:43.917320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.917250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6js\" (UniqueName: \"kubernetes.io/projected/d7999f09-dd22-470c-b935-36f29873537b-kube-api-access-wg6js\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.970132 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.970109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6js\" (UniqueName: \"kubernetes.io/projected/d7999f09-dd22-470c-b935-36f29873537b-kube-api-access-wg6js\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:43.976667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:43.976647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:44.018283 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.018264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.018379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.018300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-metrics-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.018379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.018333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-manager-config\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.018379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.018364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzb2g\" (UniqueName: \"kubernetes.io/projected/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-kube-api-access-mzb2g\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.019034 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.018942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-manager-config\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.020590 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.020574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-metrics-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.020661 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.020640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-cert\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.035940 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.035918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzb2g\" (UniqueName: \"kubernetes.io/projected/a0a7b9b1-d0e1-4f82-adad-664f44a3753b-kube-api-access-mzb2g\") pod \"lws-controller-manager-57fb49d769-6mbqf\" (UID: \"a0a7b9b1-d0e1-4f82-adad-664f44a3753b\") " pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.084438 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.084385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:44.114576 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.114551 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:44.191606 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.191554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:44.250529 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.250508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf"] Aug 05 17:25:44.252850 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:44.252826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a7b9b1_d0e1_4f82_adad_664f44a3753b.slice/crio-0c78af8a7a1f652004957bc01ee83a0a8958810f2f6832b6e8a1e14322f271af WatchSource:0}: Error finding container 0c78af8a7a1f652004957bc01ee83a0a8958810f2f6832b6e8a1e14322f271af: Status 404 returned error can't find the container with id 0c78af8a7a1f652004957bc01ee83a0a8958810f2f6832b6e8a1e14322f271af Aug 05 17:25:44.333753 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.333717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:44.427267 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.422334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:44.427267 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.425532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7999f09-dd22-470c-b935-36f29873537b-cert\") pod \"odh-model-controller-7998c67dc4-pkcpp\" (UID: \"d7999f09-dd22-470c-b935-36f29873537b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:44.582477 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.582448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:44.599642 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.599619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" event={"ID":"a0a7b9b1-d0e1-4f82-adad-664f44a3753b","Type":"ContainerStarted","Data":"0c78af8a7a1f652004957bc01ee83a0a8958810f2f6832b6e8a1e14322f271af"} Aug 05 17:25:44.707164 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.707094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pkcpp"] Aug 05 17:25:44.709583 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:44.709562 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7999f09_dd22_470c_b935_36f29873537b.slice/crio-099af0b17f45c875672be9036c6063b988a38969b4e8906493b08d44f93c443e WatchSource:0}: Error finding container 099af0b17f45c875672be9036c6063b988a38969b4e8906493b08d44f93c443e: Status 404 returned error can't find the container with id 099af0b17f45c875672be9036c6063b988a38969b4e8906493b08d44f93c443e Aug 05 17:25:44.828244 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:44.828210 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:45.235316 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:45.235223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:45.334564 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:45.334528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:45.443484 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:45.443360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:45.607733 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:45.607663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" event={"ID":"d7999f09-dd22-470c-b935-36f29873537b","Type":"ContainerStarted","Data":"099af0b17f45c875672be9036c6063b988a38969b4e8906493b08d44f93c443e"} Aug 05 17:25:47.103453 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:47.103418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:48.623609 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.623571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" event={"ID":"a0a7b9b1-d0e1-4f82-adad-664f44a3753b","Type":"ContainerStarted","Data":"b68d0b2c5940c4ffa3b1bd3ce22ef1513c4ea31f7079bf5205b636add0f17730"} Aug 05 17:25:48.624054 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.623663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:48.625039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.625015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" event={"ID":"d7999f09-dd22-470c-b935-36f29873537b","Type":"ContainerStarted","Data":"5be4263310f3a427672f4cb8ccf1448358d40329f955fe9cae484b58562243b0"} Aug 05 17:25:48.625158 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.625146 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:25:48.642808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.642772 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" podStartSLOduration=1.915156898 podStartE2EDuration="5.642761721s" totalImagesPullingTime="3.727604823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:48.64178306 +0000 UTC m=+382.072929309" watchObservedRunningTime="2026-08-05 17:25:48.642761721 +0000 UTC m=+382.073907971" Aug 05 17:25:48.660318 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:48.660282 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" podStartSLOduration=2.350078949 podStartE2EDuration="5.660269833s" totalImagesPullingTime="3.310190884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:48.659440291 +0000 UTC m=+382.090586541" watchObservedRunningTime="2026-08-05 17:25:48.660269833 +0000 UTC m=+382.091416104" Aug 05 17:25:50.768921 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:50.768881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:50.868932 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:50.868900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:51.088900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.088858 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:25:51.092188 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.092166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.096208 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.096188 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sd48x\"" Aug 05 17:25:51.096208 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.096199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 05 17:25:51.107893 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.107869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:25:51.119822 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.119800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:51.174784 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174891 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174891 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174982 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174982 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174982 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.174982 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.174980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjrv\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.175116 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.175000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.175116 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.175017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.178753 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.178734 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg"] Aug 05 17:25:51.184743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.184727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.221050 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.221030 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg"] Aug 05 17:25:51.275546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.275546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56krv\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-kube-api-access-56krv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.275740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.275985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjrv\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276472 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.276710 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.276578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.278033 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.278010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.278575 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.278556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.289830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.289810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.289921 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.289904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjrv\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcs62mh\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.377084 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377084 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377221 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377221 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377221 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377221 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56krv\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-kube-api-access-56krv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377647 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377692 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377754 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.377883 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.377867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.379380 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.379356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.379601 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.379585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.382695 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.382672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:51.386340 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.386317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.386668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.386654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56krv\" (UniqueName: \"kubernetes.io/projected/3c56e1ac-48d2-4936-92e8-32e8fcfcf61b-kube-api-access-56krv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg\" (UID: \"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.401469 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.401451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:51.494329 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.494303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:51.559451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.559399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:25:51.562127 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:51.562091 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d68501_e577_4e62_a85a_73cce3bd5ec9.slice/crio-59f84ac0afffbdef6fb1d0e666675924687ff52ff2ee760db0942d0af39f0f3a WatchSource:0}: Error finding container 59f84ac0afffbdef6fb1d0e666675924687ff52ff2ee760db0942d0af39f0f3a: Status 404 returned error can't find the container with id 59f84ac0afffbdef6fb1d0e666675924687ff52ff2ee760db0942d0af39f0f3a Aug 05 17:25:51.636755 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.636677 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg"] Aug 05 17:25:51.639717 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.639690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" event={"ID":"47d68501-e577-4e62-a85a-73cce3bd5ec9","Type":"ContainerStarted","Data":"59f84ac0afffbdef6fb1d0e666675924687ff52ff2ee760db0942d0af39f0f3a"} Aug 05 17:25:51.639883 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:51.639862 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c56e1ac_48d2_4936_92e8_32e8fcfcf61b.slice/crio-3e29c8fe65cbe2dc1bffad9b0cd245978bf0847f0540222f90e7844800fc382d WatchSource:0}: Error finding container 3e29c8fe65cbe2dc1bffad9b0cd245978bf0847f0540222f90e7844800fc382d: Status 404 returned error can't find the container with id 3e29c8fe65cbe2dc1bffad9b0cd245978bf0847f0540222f90e7844800fc382d Aug 05 17:25:51.833735 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:51.833701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:52.154013 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.153982 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h"] Aug 05 17:25:52.158564 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.158549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.161401 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.161383 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 05 17:25:52.161596 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.161583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 05 17:25:52.161666 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.161651 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j989w\"" Aug 05 17:25:52.161844 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.161831 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 05 17:25:52.171389 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.171368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h"] Aug 05 17:25:52.189177 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189276 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1376d3c1-28e8-40e6-91d6-cf88e69f6327-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.189405 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.189327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qzm\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-kube-api-access-j2qzm\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.290319 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.290289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.290801 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.290740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qzm\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-kube-api-access-j2qzm\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.291418 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.291392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.291521 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.291442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.291521 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.291501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.291630 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.291584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.291630 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.291613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1376d3c1-28e8-40e6-91d6-cf88e69f6327-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.295231 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.294929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.296797 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.296764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1376d3c1-28e8-40e6-91d6-cf88e69f6327-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.296944 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.296832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.296944 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.296858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.297144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.297088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1376d3c1-28e8-40e6-91d6-cf88e69f6327-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.300320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.300284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qzm\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-kube-api-access-j2qzm\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.304081 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.304049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1376d3c1-28e8-40e6-91d6-cf88e69f6327-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zjc2h\" (UID: \"1376d3c1-28e8-40e6-91d6-cf88e69f6327\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.467700 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.467626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:52.647561 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.647528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" event={"ID":"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b","Type":"ContainerStarted","Data":"3e29c8fe65cbe2dc1bffad9b0cd245978bf0847f0540222f90e7844800fc382d"} Aug 05 17:25:52.883139 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:52.883096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h"] Aug 05 17:25:52.885992 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:25:52.885942 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1376d3c1_28e8_40e6_91d6_cf88e69f6327.slice/crio-7b0813d266c7c370cad8d0c899e519f5a84c5d99270c2aa6b0b099969c2dc579 WatchSource:0}: Error finding container 7b0813d266c7c370cad8d0c899e519f5a84c5d99270c2aa6b0b099969c2dc579: Status 404 returned error can't find the container with id 7b0813d266c7c370cad8d0c899e519f5a84c5d99270c2aa6b0b099969c2dc579 Aug 05 17:25:53.653297 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:53.653259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" event={"ID":"1376d3c1-28e8-40e6-91d6-cf88e69f6327","Type":"ContainerStarted","Data":"7b0813d266c7c370cad8d0c899e519f5a84c5d99270c2aa6b0b099969c2dc579"} Aug 05 17:25:54.391572 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.391532 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.391862 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.391603 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.391862 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.391636 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.398417 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.398394 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.398477 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.398449 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.398477 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.398472 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:54.660546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.660436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" event={"ID":"3c56e1ac-48d2-4936-92e8-32e8fcfcf61b","Type":"ContainerStarted","Data":"7a397978f273307f362354c608d50a5e480766cb6d187f355200edf5489b8a1e"} Aug 05 17:25:54.662509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.662483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" event={"ID":"47d68501-e577-4e62-a85a-73cce3bd5ec9","Type":"ContainerStarted","Data":"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a"} Aug 05 17:25:54.730084 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.730028 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" podStartSLOduration=0.980416682 podStartE2EDuration="3.73000827s" totalImagesPullingTime="2.749591588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:54.694368843 +0000 UTC m=+388.125515094" watchObservedRunningTime="2026-08-05 17:25:54.73000827 +0000 UTC m=+388.161154521" Aug 05 17:25:54.731887 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.731841 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" podStartSLOduration=0.898363791 podStartE2EDuration="3.731827649s" totalImagesPullingTime="2.833463858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:54.728734329 +0000 UTC m=+388.159880590" watchObservedRunningTime="2026-08-05 17:25:54.731827649 +0000 UTC m=+388.162973899" Aug 05 17:25:54.757823 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.757797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:54.850432 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:54.850372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:55.197378 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.197349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:55.314607 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.314561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:55.402025 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.401950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:25:55.403295 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.403260 2564 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Aug 05 17:25:55.403394 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.403355 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Aug 05 17:25:55.495416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.495338 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:55.500930 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.500909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:55.668127 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.668100 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:55.669147 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.669122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg" Aug 05 17:25:55.736585 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:55.736552 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:25:56.056302 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.056269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:56.154893 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.154858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:56.249466 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.249431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:25:56.281870 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.281836 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:56.281970 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.281905 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:25:56.402364 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.402333 2564 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Aug 05 17:25:56.402695 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.402395 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Aug 05 17:25:56.672995 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.672892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" event={"ID":"1376d3c1-28e8-40e6-91d6-cf88e69f6327","Type":"ContainerStarted","Data":"c4f049fa11e2c42aa28e22bf069a58dcb4aefd91067104498bce36dbaa761262"} Aug 05 17:25:56.673185 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.673168 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:56.709595 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:56.709549 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" podStartSLOduration=1.316691008 podStartE2EDuration="4.709534761s" totalImagesPullingTime="3.392843753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:56.708659833 +0000 UTC m=+390.139806083" watchObservedRunningTime="2026-08-05 17:25:56.709534761 +0000 UTC m=+390.140681009" Aug 05 17:25:57.402640 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:57.402603 2564 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Aug 05 17:25:57.403090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:57.402661 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Aug 05 17:25:57.677439 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:57.677350 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" containerID="cri-o://9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a" gracePeriod=30 Aug 05 17:25:57.679075 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:57.679055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zjc2h" Aug 05 17:25:59.631314 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:59.631286 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57fb49d769-6mbqf" Aug 05 17:25:59.678435 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:25:59.678409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pkcpp" Aug 05 17:26:02.921965 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.921924 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:26:02.986425 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986358 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986425 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986394 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986425 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfjrv\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986522 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986566 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986615 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:26:02.986680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket\") pod \"47d68501-e577-4e62-a85a-73cce3bd5ec9\" (UID: \"47d68501-e577-4e62-a85a-73cce3bd5ec9\") " Aug 05 17:26:02.987021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:26:02.987021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986873 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:26:02.987021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986894 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:26:02.987021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986935 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:02.987021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.986991 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data" (OuterVolumeSpecName: "istio-data") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:26:02.987251 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.987049 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d68501-e577-4e62-a85a-73cce3bd5ec9-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:02.987251 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.987072 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:02.989029 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.989002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:26:02.989143 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.989111 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token" (OuterVolumeSpecName: "istio-token") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:26:02.989192 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.989154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv" (OuterVolumeSpecName: "kube-api-access-hfjrv") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "kube-api-access-hfjrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:26:02.989327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:02.989312 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47d68501-e577-4e62-a85a-73cce3bd5ec9" (UID: "47d68501-e577-4e62-a85a-73cce3bd5ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:26:03.088403 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088381 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjrv\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-kube-api-access-hfjrv\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.088403 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088402 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.088542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088412 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.088542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088425 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.088542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088435 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d68501-e577-4e62-a85a-73cce3bd5ec9-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.088542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.088444 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d68501-e577-4e62-a85a-73cce3bd5ec9-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:26:03.703210 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.703183 2564 generic.go:356] "Generic (PLEG): container finished" podID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerID="9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a" exitCode=0 Aug 05 17:26:03.703361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.703226 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" Aug 05 17:26:03.703361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.703274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" event={"ID":"47d68501-e577-4e62-a85a-73cce3bd5ec9","Type":"ContainerDied","Data":"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a"} Aug 05 17:26:03.703361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.703324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh" event={"ID":"47d68501-e577-4e62-a85a-73cce3bd5ec9","Type":"ContainerDied","Data":"59f84ac0afffbdef6fb1d0e666675924687ff52ff2ee760db0942d0af39f0f3a"} Aug 05 17:26:03.703361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.703346 2564 scope.go:117] "RemoveContainer" containerID="9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a" Aug 05 17:26:03.711509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.711485 2564 scope.go:117] "RemoveContainer" containerID="9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a" Aug 05 17:26:03.711701 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:26:03.711684 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a\": container with ID starting with 9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a not found: ID does not exist" containerID="9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a" Aug 05 17:26:03.711757 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.711708 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a"} err="failed to get container status \"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a\": rpc error: code = NotFound desc = could not find container \"9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a\": container with ID starting with 9980e1687fcb2aa1fc96545832dfe8f3920fd8254d84a9ad8d4f9f938b6fe68a not found: ID does not exist" Aug 05 17:26:03.721462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.721444 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:26:03.725390 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:03.725371 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcs62mh"] Aug 05 17:26:05.038114 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:05.038084 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" path="/var/lib/kubelet/pods/47d68501-e577-4e62-a85a-73cce3bd5ec9/volumes" Aug 05 17:26:28.474165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:28.474128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:28.628335 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:28.628307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:28.697708 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:28.697680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:29.224702 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:29.224668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:29.324839 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:29.324811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:39.310337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:39.310302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:39.408278 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:39.408239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:44.357294 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:44.357251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:44.456600 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:44.456566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:46.147659 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:46.147614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:26:58.308307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:26:58.308228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:00.914667 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:00.914626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:01.304452 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:01.304423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:01.707153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:01.707062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:04.545439 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:04.545406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:04.701271 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:04.701237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:05.290432 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:05.290390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:05.727179 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:05.727143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:05.838587 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:05.838552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:07.184887 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:07.184855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:07.271939 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:07.271906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:07.939081 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:07.939042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:08.565096 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:08.565060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:10.976036 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:10.976003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:11.110257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:11.110218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:14.249892 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:14.249858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:14.544890 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:14.544851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:15.113030 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:15.112999 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:15.218522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:15.218485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:15.351194 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:15.351163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:15.452519 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:15.452432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:16.349546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:16.349516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:16.485524 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:16.485492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:23.865598 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:23.865546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:23.964819 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:23.964781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:24.085609 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:24.085582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:24.249145 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:24.249067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:26.293752 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:26.293713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:26.513617 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:26.513591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:30.606318 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:30.606281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:30.803523 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:30.803495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:34.248361 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:34.248327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:34.348297 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:34.348259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:34.449205 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:34.449174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:34.510818 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:34.510745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:35.449194 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:35.449165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:35.517137 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:35.517107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:43.463393 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:43.463361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:43.562085 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:43.562055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:45.937217 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:45.937187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:46.038076 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:46.038045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:55.271217 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.271179 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb"] Aug 05 17:27:55.271653 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.271627 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" Aug 05 17:27:55.271653 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.271652 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" Aug 05 17:27:55.271819 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.271739 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d68501-e577-4e62-a85a-73cce3bd5ec9" containerName="istio-proxy" Aug 05 17:27:55.279706 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.279673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.283198 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.283168 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2fhbc\"" Aug 05 17:27:55.286019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.285997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb"] Aug 05 17:27:55.313336 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.313308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:55.437045 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wgl\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-kube-api-access-q5wgl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437177 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437177 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437177 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437325 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437325 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437325 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437325 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.437325 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.437265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.438599 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.438574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:55.537830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.537830 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.537979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wgl\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-kube-api-access-q5wgl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538039 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538373 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538373 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538373 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538503 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.538748 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.538731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.540259 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.540240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.540463 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.540448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.545887 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.545865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.546148 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.546125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wgl\" (UniqueName: \"kubernetes.io/projected/7ef34bc7-0b58-4eca-88f4-52fcad58714d-kube-api-access-q5wgl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p2lzb\" (UID: \"7ef34bc7-0b58-4eca-88f4-52fcad58714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.592723 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.592701 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:55.723430 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.723406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb"] Aug 05 17:27:55.725219 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:27:55.725190 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef34bc7_0b58_4eca_88f4_52fcad58714d.slice/crio-bfd67cc21b221084aaa62e88c429d66bee7403f9ef8b46b0711807efcda5a4e0 WatchSource:0}: Error finding container bfd67cc21b221084aaa62e88c429d66bee7403f9ef8b46b0711807efcda5a4e0: Status 404 returned error can't find the container with id bfd67cc21b221084aaa62e88c429d66bee7403f9ef8b46b0711807efcda5a4e0 Aug 05 17:27:55.727234 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.727197 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:27:55.727346 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.727277 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:27:55.727346 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:55.727324 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:27:56.159917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:56.159888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" event={"ID":"7ef34bc7-0b58-4eca-88f4-52fcad58714d","Type":"ContainerStarted","Data":"ef38463510ca57f8299e6c86b4a5c8ffc8aca704c5b3c371cd5d6ae20aebe9f7"} Aug 05 17:27:56.160083 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:56.159937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" event={"ID":"7ef34bc7-0b58-4eca-88f4-52fcad58714d","Type":"ContainerStarted","Data":"bfd67cc21b221084aaa62e88c429d66bee7403f9ef8b46b0711807efcda5a4e0"} Aug 05 17:27:56.179270 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:56.179221 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podStartSLOduration=1.179204481 podStartE2EDuration="1.179204481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:27:56.17908758 +0000 UTC m=+509.610233829" watchObservedRunningTime="2026-08-05 17:27:56.179204481 +0000 UTC m=+509.610350733" Aug 05 17:27:56.593539 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:56.593514 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:56.597670 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:56.597649 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:57.163936 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:57.163909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:57.164782 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:57.164765 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:27:57.289458 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:57.289423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:57.394690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:57.394662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:58.101199 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:58.101165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:27:58.187090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:27:58.187061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:09.255737 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:09.255681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:09.316840 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:09.316796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:13.497622 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:13.497591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:13.587878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:13.587845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:13.976309 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:13.976260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:22.999278 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:22.999197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:23.070379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:23.070354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:26.446231 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:26.446194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:26.545607 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:26.545578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:35.750826 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:35.750776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:35.851019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:35.850981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:35.933200 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:35.933159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:36.792331 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:36.792298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:40.358247 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:40.358213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:40.458785 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:40.458754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:28:40.869905 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:28:40.869873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:09.740893 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:09.740863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:09.806784 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:09.806757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:11.700503 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:11.700457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:11.801823 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:11.801790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:13.223723 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:13.223689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:13.688401 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:13.688371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:13.788053 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:13.788025 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:13.934009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:13.933980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:23.522522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.522492 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerID="ef38463510ca57f8299e6c86b4a5c8ffc8aca704c5b3c371cd5d6ae20aebe9f7" exitCode=0 Aug 05 17:29:23.522915 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.522574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" event={"ID":"7ef34bc7-0b58-4eca-88f4-52fcad58714d","Type":"ContainerDied","Data":"ef38463510ca57f8299e6c86b4a5c8ffc8aca704c5b3c371cd5d6ae20aebe9f7"} Aug 05 17:29:23.522975 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.522933 2564 scope.go:117] "RemoveContainer" containerID="ef38463510ca57f8299e6c86b4a5c8ffc8aca704c5b3c371cd5d6ae20aebe9f7" Aug 05 17:29:23.523892 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.523866 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:29:23.523946 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.523937 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:29:23.524000 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:23.523985 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:29:24.359220 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.359176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:24.457585 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.457552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:24.528340 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.528307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" event={"ID":"7ef34bc7-0b58-4eca-88f4-52fcad58714d","Type":"ContainerStarted","Data":"24bc72be576e4c718c76096f03f7c4cc9b34341d0e487f8c507f15c507c3227d"} Aug 05 17:29:24.593917 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.593879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:29:24.594131 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.594033 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p2lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Aug 05 17:29:24.594131 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:24.594090 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podUID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Aug 05 17:29:25.593857 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:25.593731 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p2lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Aug 05 17:29:25.597976 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:25.594616 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podUID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Aug 05 17:29:26.593346 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:26.593307 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p2lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Aug 05 17:29:26.593547 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:26.593375 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podUID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Aug 05 17:29:27.164346 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:27.164315 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:29:27.593809 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:27.593770 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p2lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Aug 05 17:29:27.594017 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:27.593833 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podUID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Aug 05 17:29:28.593220 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:28.593188 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p2lzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Aug 05 17:29:28.593753 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:28.593726 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" podUID="7ef34bc7-0b58-4eca-88f4-52fcad58714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Aug 05 17:29:29.597358 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:29.597328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:29:29.598453 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:29.598431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p2lzb" Aug 05 17:29:35.767497 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.767466 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck"] Aug 05 17:29:35.769622 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.769604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.773373 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.773354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 17:29:35.773494 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.773387 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 17:29:35.773494 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.773402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 05 17:29:35.773494 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.773413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5llc\"" Aug 05 17:29:35.780206 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.780188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck"] Aug 05 17:29:35.795544 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.795516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:35.891815 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.891912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.891912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.891912 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vg5g\" (UniqueName: \"kubernetes.io/projected/5910bc30-655c-42ee-8133-f1b215f3a0f2-kube-api-access-4vg5g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.892058 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.892058 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.891939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5910bc30-655c-42ee-8133-f1b215f3a0f2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993221 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vg5g\" (UniqueName: \"kubernetes.io/projected/5910bc30-655c-42ee-8133-f1b215f3a0f2-kube-api-access-4vg5g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5910bc30-655c-42ee-8133-f1b215f3a0f2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993352 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993516 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993516 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993662 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993752 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.993806 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.993758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.995562 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.995542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5910bc30-655c-42ee-8133-f1b215f3a0f2-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:35.995828 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:35.995813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5910bc30-655c-42ee-8133-f1b215f3a0f2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:36.001132 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:36.001110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vg5g\" (UniqueName: \"kubernetes.io/projected/5910bc30-655c-42ee-8133-f1b215f3a0f2-kube-api-access-4vg5g\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck\" (UID: \"5910bc30-655c-42ee-8133-f1b215f3a0f2\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:36.080496 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:36.080473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:36.211141 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:36.210997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck"] Aug 05 17:29:36.213195 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:29:36.213171 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5910bc30_655c_42ee_8133_f1b215f3a0f2.slice/crio-8c87914316f66cbccc5b520e2576d9f52d7e60c24ac9b01c7792995b2a0a0ce4 WatchSource:0}: Error finding container 8c87914316f66cbccc5b520e2576d9f52d7e60c24ac9b01c7792995b2a0a0ce4: Status 404 returned error can't find the container with id 8c87914316f66cbccc5b520e2576d9f52d7e60c24ac9b01c7792995b2a0a0ce4 Aug 05 17:29:36.576574 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:36.576539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" event={"ID":"5910bc30-655c-42ee-8133-f1b215f3a0f2","Type":"ContainerStarted","Data":"8c87914316f66cbccc5b520e2576d9f52d7e60c24ac9b01c7792995b2a0a0ce4"} Aug 05 17:29:37.582004 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:37.581908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" event={"ID":"5910bc30-655c-42ee-8133-f1b215f3a0f2","Type":"ContainerStarted","Data":"10ca50d46772d0d25adb9890fdb0cd683cbe2b868fdd9720dc4ebda0108c1fc5"} Aug 05 17:29:37.601965 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:37.601919 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" podStartSLOduration=1.512497957 podStartE2EDuration="2.601907255s" totalImagesPullingTime="1.089409298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:37.601135435 +0000 UTC m=+611.032281688" watchObservedRunningTime="2026-08-05 17:29:37.601907255 +0000 UTC m=+611.033053528" Aug 05 17:29:44.371074 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:44.371040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:46.081527 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:46.081500 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:46.081870 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:46.081551 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:46.093315 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:46.093295 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:46.627489 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:46.627464 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck" Aug 05 17:29:50.164173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.164098 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb"] Aug 05 17:29:50.167654 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.167639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.170330 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.170303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 05 17:29:50.177019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.176993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb"] Aug 05 17:29:50.195046 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.195019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:50.296414 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gmq\" (UniqueName: \"kubernetes.io/projected/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-kube-api-access-t5gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.296414 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.296619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.296619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.296619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.296619 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.296582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397233 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397345 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gmq\" (UniqueName: \"kubernetes.io/projected/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-kube-api-access-t5gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397397 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397475 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397525 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397579 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.397776 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.397758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.399505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.399487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.399821 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.399801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.406686 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.406666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gmq\" (UniqueName: \"kubernetes.io/projected/e3d163fe-9c4b-4fcb-9ce6-95beaa988f65-kube-api-access-t5gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ltqdb\" (UID: \"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.478113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.478064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:29:50.606531 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.606502 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb"] Aug 05 17:29:50.608708 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:29:50.608686 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d163fe_9c4b_4fcb_9ce6_95beaa988f65.slice/crio-527524a7642714614fa7e5d022fd905f9d749fc242bb80267c13003b68f00ebe WatchSource:0}: Error finding container 527524a7642714614fa7e5d022fd905f9d749fc242bb80267c13003b68f00ebe: Status 404 returned error can't find the container with id 527524a7642714614fa7e5d022fd905f9d749fc242bb80267c13003b68f00ebe Aug 05 17:29:50.633862 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:50.633830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" event={"ID":"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65","Type":"ContainerStarted","Data":"527524a7642714614fa7e5d022fd905f9d749fc242bb80267c13003b68f00ebe"} Aug 05 17:29:51.641097 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:51.641063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" event={"ID":"e3d163fe-9c4b-4fcb-9ce6-95beaa988f65","Type":"ContainerStarted","Data":"9cb013a1d4da0bfea57f52b65271a35f72af05ece0a2c41cf383b68246d2ac44"} Aug 05 17:29:51.663092 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:51.663040 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" podStartSLOduration=1.4512903160000001 podStartE2EDuration="1.663020091s" totalImagesPullingTime="211.729775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:51.660151313 +0000 UTC m=+625.091297553" watchObservedRunningTime="2026-08-05 17:29:51.663020091 +0000 UTC m=+625.094166342" Aug 05 17:29:55.353498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:55.353465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:29:55.449358 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:29:55.449321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:30:00.141697 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.141666 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5"] Aug 05 17:30:00.145747 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.145731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" Aug 05 17:30:00.148420 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.148403 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bz9qt\"" Aug 05 17:30:00.152281 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.152261 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5"] Aug 05 17:30:00.157105 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.157089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" Aug 05 17:30:00.281362 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.281183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5"] Aug 05 17:30:00.283429 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:30:00.283393 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a7978c_ad50_4e29_9a1d_3aaf56212930.slice/crio-1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4 WatchSource:0}: Error finding container 1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4: Status 404 returned error can't find the container with id 1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4 Aug 05 17:30:00.478441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.478348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:30:00.478603 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.478448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:30:00.490122 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.490103 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:30:00.678621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.678591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" event={"ID":"20a7978c-ad50-4e29-9a1d-3aaf56212930","Type":"ContainerStarted","Data":"1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4"} Aug 05 17:30:00.689348 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:00.689327 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ltqdb" Aug 05 17:30:03.694025 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:03.693926 2564 generic.go:356] "Generic (PLEG): container finished" podID="20a7978c-ad50-4e29-9a1d-3aaf56212930" containerID="a0fc795253cfabd7d2160bb2005f358fb0d68bfa26c54b36307cd1429d9864ff" exitCode=0 Aug 05 17:30:03.694025 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:03.694015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" event={"ID":"20a7978c-ad50-4e29-9a1d-3aaf56212930","Type":"ContainerDied","Data":"a0fc795253cfabd7d2160bb2005f358fb0d68bfa26c54b36307cd1429d9864ff"} Aug 05 17:30:04.775291 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.775259 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt"] Aug 05 17:30:04.779297 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.779277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.781906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.781888 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 05 17:30:04.788062 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.788039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt"] Aug 05 17:30:04.808792 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.808762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.808867 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.808849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.808923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.808880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.808923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.808904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fkb\" (UniqueName: \"kubernetes.io/projected/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-kube-api-access-p6fkb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.809033 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.808981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.809033 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.809007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.826369 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.826352 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" Aug 05 17:30:04.888066 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.888029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:30:04.910078 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910206 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910206 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910206 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fkb\" (UniqueName: \"kubernetes.io/projected/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-kube-api-access-p6fkb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910206 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910376 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910567 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910644 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.910702 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.910670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.912370 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.912354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.912744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.912726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:04.920863 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:04.920843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fkb\" (UniqueName: \"kubernetes.io/projected/ef607391-12a1-40fc-8fa0-10bfb5bfb38e-kube-api-access-p6fkb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt\" (UID: \"ef607391-12a1-40fc-8fa0-10bfb5bfb38e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:05.092082 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.092050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:05.218980 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.218943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt"] Aug 05 17:30:05.221072 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:30:05.221041 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef607391_12a1_40fc_8fa0_10bfb5bfb38e.slice/crio-116979183bf678c6e7f7559f994bd53cc8a54a40eac2283b05af0cd680ab4cf9 WatchSource:0}: Error finding container 116979183bf678c6e7f7559f994bd53cc8a54a40eac2283b05af0cd680ab4cf9: Status 404 returned error can't find the container with id 116979183bf678c6e7f7559f994bd53cc8a54a40eac2283b05af0cd680ab4cf9 Aug 05 17:30:05.703254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.703176 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" Aug 05 17:30:05.703254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.703212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5" event={"ID":"20a7978c-ad50-4e29-9a1d-3aaf56212930","Type":"ContainerDied","Data":"1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4"} Aug 05 17:30:05.703469 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.703261 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc233f809accbe93e5d28b9a752e3eea9892d92553fdaa3d23f55f29c8f61b4" Aug 05 17:30:05.704714 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.704691 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" event={"ID":"ef607391-12a1-40fc-8fa0-10bfb5bfb38e","Type":"ContainerStarted","Data":"dc388c2c7e368bfd6cfaf3cfcbf7a53fae6231fb64cafe3bfd46306652bf8e0e"} Aug 05 17:30:05.704833 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.704727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" event={"ID":"ef607391-12a1-40fc-8fa0-10bfb5bfb38e","Type":"ContainerStarted","Data":"116979183bf678c6e7f7559f994bd53cc8a54a40eac2283b05af0cd680ab4cf9"} Aug 05 17:30:05.723079 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:05.723037 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" podStartSLOduration=1.499327212 podStartE2EDuration="1.723024402s" totalImagesPullingTime="223.69719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:05.721342929 +0000 UTC m=+639.152489179" watchObservedRunningTime="2026-08-05 17:30:05.723024402 +0000 UTC m=+639.154170651" Aug 05 17:30:06.279440 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:06.279405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:30:11.710298 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:11.710261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:30:15.093154 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:15.093121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:15.093512 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:15.093182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:15.105333 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:15.105310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:15.757604 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:15.757575 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt" Aug 05 17:30:21.673120 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:21.673082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:30:57.159134 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:30:57.159079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:31:14.220846 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:31:14.220812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:31:36.321235 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:31:36.321148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:31:36.421182 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:31:36.421152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:31:37.482250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:31:37.482222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:31:37.586905 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:31:37.586864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:32:34.682212 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:32:34.682178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:32:34.748971 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:32:34.748923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:43.042102 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:33:43.042066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:44.791008 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:33:44.790941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:34:49.100884 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:34:49.100846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:34:49.200858 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:34:49.200826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:34:49.341048 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:34:49.341012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:34:55.049800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:34:55.049763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:34:56.699683 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:34:56.699646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:35:01.108212 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:35:01.108177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:35:01.226974 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:35:01.226889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:36:51.939527 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:36:51.939491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:36:52.024145 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:36:52.024106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:36:56.310419 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:36:56.310374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:37:16.138708 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:37:16.138667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:37:16.241238 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:37:16.241208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:40:15.378686 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:40:15.378657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:42:53.261442 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:42:53.261409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:45:00.140647 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.140566 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb"] Aug 05 17:45:00.141130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.140902 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a7978c-ad50-4e29-9a1d-3aaf56212930" containerName="cleanup" Aug 05 17:45:00.141130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.140912 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a7978c-ad50-4e29-9a1d-3aaf56212930" containerName="cleanup" Aug 05 17:45:00.141130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.141008 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a7978c-ad50-4e29-9a1d-3aaf56212930" containerName="cleanup" Aug 05 17:45:00.143932 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.143916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" Aug 05 17:45:00.146532 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.146513 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bz9qt\"" Aug 05 17:45:00.153064 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.153046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" Aug 05 17:45:00.158024 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.158002 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb"] Aug 05 17:45:00.294689 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.294629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb"] Aug 05 17:45:00.297061 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:45:00.297034 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c0dfd6_7c12_4c0d_b52a_ec499b990280.slice/crio-47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295 WatchSource:0}: Error finding container 47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295: Status 404 returned error can't find the container with id 47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295 Aug 05 17:45:00.567355 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.567324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" event={"ID":"83c0dfd6-7c12-4c0d-b52a-ec499b990280","Type":"ContainerStarted","Data":"ba539125a3a6060637e98c39b1da1b62cbbb4aa4aac6370ae1809ad934cc5a27"} Aug 05 17:45:00.567546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.567381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" event={"ID":"83c0dfd6-7c12-4c0d-b52a-ec499b990280","Type":"ContainerStarted","Data":"47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295"} Aug 05 17:45:00.583232 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:00.583180 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" podStartSLOduration=0.583163617 podStartE2EDuration="583.163617ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:00.581925848 +0000 UTC m=+1534.013072098" watchObservedRunningTime="2026-08-05 17:45:00.583163617 +0000 UTC m=+1534.014309866" Aug 05 17:45:01.573569 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:01.573533 2564 generic.go:356] "Generic (PLEG): container finished" podID="83c0dfd6-7c12-4c0d-b52a-ec499b990280" containerID="ba539125a3a6060637e98c39b1da1b62cbbb4aa4aac6370ae1809ad934cc5a27" exitCode=0 Aug 05 17:45:01.573569 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:01.573574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" event={"ID":"83c0dfd6-7c12-4c0d-b52a-ec499b990280","Type":"ContainerDied","Data":"ba539125a3a6060637e98c39b1da1b62cbbb4aa4aac6370ae1809ad934cc5a27"} Aug 05 17:45:02.709021 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:02.708994 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" Aug 05 17:45:03.588566 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:03.588525 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" event={"ID":"83c0dfd6-7c12-4c0d-b52a-ec499b990280","Type":"ContainerDied","Data":"47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295"} Aug 05 17:45:03.588740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:03.588584 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47896b97d922be40a9ddae2c11dc5643962feb9fae9d5b2e9f610dc52153f295" Aug 05 17:45:03.588740 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:03.588550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-7d9tb" Aug 05 17:45:14.596407 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:14.596378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:45:23.124602 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:45:23.124571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:46:00.921433 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:46:00.921385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:46:02.427370 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:46:02.427337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:46:04.731717 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:46:04.731680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:46:04.832449 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:46:04.832409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:23.572188 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:23.572114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:49.366933 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:49.366842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:52.389075 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:52.389017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:52.492204 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:52.492170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:52.817508 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:52.817478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:58.442424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.442391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:58.681200 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.681162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:58.890868 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.890841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:58.938214 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.938185 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:49:58.938537 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.938526 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c0dfd6-7c12-4c0d-b52a-ec499b990280" containerName="cleanup" Aug 05 17:49:58.938586 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.938539 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c0dfd6-7c12-4c0d-b52a-ec499b990280" containerName="cleanup" Aug 05 17:49:58.938641 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.938601 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c0dfd6-7c12-4c0d-b52a-ec499b990280" containerName="cleanup" Aug 05 17:49:58.941542 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.941520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:58.943873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.943844 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fbbe1196-openshift-default-dockercfg-s628h\"" Aug 05 17:49:58.964442 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.964421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:49:58.966514 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:58.966495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:59.085491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085491 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9h4\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.085742 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.086089 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.085760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187028 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.186925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187028 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9h4\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187257 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187600 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187600 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187600 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187576 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187846 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.187923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.187795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.189403 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.189382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.189755 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.189738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.193106 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.193077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:49:59.195007 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.194982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.195266 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.195251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9h4\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4\") pod \"e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.251113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.251093 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:49:59.387327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.387300 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:49:59.388833 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:49:59.388808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83858385_7231_4a2d_ae1e_f4826638fa83.slice/crio-4e52b69f056a5588f21f2c321d5aaa8dc42642b381ad6ac1faaab9159a99f073 WatchSource:0}: Error finding container 4e52b69f056a5588f21f2c321d5aaa8dc42642b381ad6ac1faaab9159a99f073: Status 404 returned error can't find the container with id 4e52b69f056a5588f21f2c321d5aaa8dc42642b381ad6ac1faaab9159a99f073 Aug 05 17:49:59.390906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.390875 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:49:59.391043 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.390950 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:49:59.391043 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.391010 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:49:59.893991 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.893933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerStarted","Data":"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d"} Aug 05 17:49:59.894431 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.894016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerStarted","Data":"4e52b69f056a5588f21f2c321d5aaa8dc42642b381ad6ac1faaab9159a99f073"} Aug 05 17:49:59.916136 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:49:59.916088 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podStartSLOduration=1.916073292 podStartE2EDuration="1.916073292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:49:59.913741268 +0000 UTC m=+1833.344887538" watchObservedRunningTime="2026-08-05 17:49:59.916073292 +0000 UTC m=+1833.347219954" Aug 05 17:50:00.251407 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:00.251347 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:00.255877 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:00.255856 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:00.899444 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:00.899419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:00.900384 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:00.900367 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:01.565189 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:01.565147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:01.667930 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:01.667898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:02.699178 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:02.699146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:03.001421 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:03.001323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:03.301030 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:03.301001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:03.374876 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:03.374850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:15.901455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.901420 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:15.901844 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.901468 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:15.965646 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.965616 2564 generic.go:356] "Generic (PLEG): container finished" podID="83858385-7231-4a2d-ae1e-f4826638fa83" containerID="3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d" exitCode=0 Aug 05 17:50:15.965778 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.965658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerDied","Data":"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d"} Aug 05 17:50:15.966037 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.966023 2564 scope.go:117] "RemoveContainer" containerID="3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d" Aug 05 17:50:15.966909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.966882 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:15.966971 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.966945 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:15.967012 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:15.967002 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:16.971327 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:16.971297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerStarted","Data":"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15"} Aug 05 17:50:17.251991 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:17.251873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:17.252144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:17.252085 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:17.252195 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:17.252148 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:18.251534 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:18.251500 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:18.251923 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:18.251563 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:19.252111 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:19.252074 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:19.252565 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:19.252131 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:20.251855 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:20.251817 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:20.252073 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:20.251874 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:21.252150 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:21.252113 2564 patch_prober.go:28] interesting pod/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Aug 05 17:50:21.252528 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:21.252164 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Aug 05 17:50:22.255507 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:22.255478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:22.255895 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:22.255823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:22.256588 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:22.256573 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:36.050597 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:36.050560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:43.376184 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.376139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:43.476188 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.476136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:43.555851 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.555827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:43.693315 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.693232 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:50:43.693766 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.693733 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" containerID="cri-o://11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15" gracePeriod=30 Aug 05 17:50:43.716763 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.716724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:43.951042 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:43.950941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:44.173731 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.173705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:44.283602 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.283535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:44.399466 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.399438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:44.662978 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.662936 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:50:44.665288 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.665272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.667943 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.667921 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-41ad7fbb-gw-openshift-default-dockercfg-62jtl\"" Aug 05 17:50:44.679763 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.679743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:50:44.688481 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.688450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:44.694766 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.694742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.694838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.694787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.694901 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.694884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.694942 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.694916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.695057 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.695041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.695097 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.695066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.695097 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.695082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.695175 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.695100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4mt\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.695175 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.695123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796142 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796142 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4mt\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796416 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796732 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796732 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796796 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796882 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.796979 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.796899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.798859 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.798838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.799070 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.799054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.805246 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.805224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.806465 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.806436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4mt\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt\") pod \"e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:44.974888 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:44.974800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:45.107303 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:45.107282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:50:45.109100 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:50:45.109073 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4674678e_5e8e_4f29_995a_cb985b5b562e.slice/crio-b635cb12c30434070bd23ce44fc14c3b34720e4940d04f1598d154e61620a43e WatchSource:0}: Error finding container b635cb12c30434070bd23ce44fc14c3b34720e4940d04f1598d154e61620a43e: Status 404 returned error can't find the container with id b635cb12c30434070bd23ce44fc14c3b34720e4940d04f1598d154e61620a43e Aug 05 17:50:45.111047 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:45.111018 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:45.111113 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:45.111083 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:45.111148 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:45.111110 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:50:45.219922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:45.219888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:46.101948 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:46.101914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerStarted","Data":"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720"} Aug 05 17:50:46.102371 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:46.102002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerStarted","Data":"b635cb12c30434070bd23ce44fc14c3b34720e4940d04f1598d154e61620a43e"} Aug 05 17:50:46.127533 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:46.127482 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podStartSLOduration=2.127465909 podStartE2EDuration="2.127465909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:50:46.126524374 +0000 UTC m=+1879.557670623" watchObservedRunningTime="2026-08-05 17:50:46.127465909 +0000 UTC m=+1879.558612158" Aug 05 17:50:46.975376 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:46.975338 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:46.980231 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:46.980209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:47.106392 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:47.106361 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:47.107264 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:47.107243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:50:47.596236 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:47.596202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:47.731307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:47.731219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:47.992260 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:47.992175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:48.371300 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:48.371267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:48.528171 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:48.528136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:50:48.933419 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:48.933396 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:49.033682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033589 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033647 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033697 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033715 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk9h4\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.033983 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.033826 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert\") pod \"83858385-7231-4a2d-ae1e-f4826638fa83\" (UID: \"83858385-7231-4a2d-ae1e-f4826638fa83\") " Aug 05 17:50:49.034271 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:50:49.034271 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034049 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data" (OuterVolumeSpecName: "istio-data") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:50:49.034271 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034237 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:50:49.034424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034276 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.034424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034296 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.034424 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034409 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:50:49.034588 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.034432 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:50:49.036355 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.036325 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:50:49.036584 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.036557 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:50:49.036584 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.036580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token" (OuterVolumeSpecName: "istio-token") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:50:49.036722 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.036650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4" (OuterVolumeSpecName: "kube-api-access-nk9h4") pod "83858385-7231-4a2d-ae1e-f4826638fa83" (UID: "83858385-7231-4a2d-ae1e-f4826638fa83"). InnerVolumeSpecName "kube-api-access-nk9h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:50:49.117226 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.117190 2564 generic.go:356] "Generic (PLEG): container finished" podID="83858385-7231-4a2d-ae1e-f4826638fa83" containerID="11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15" exitCode=0 Aug 05 17:50:49.117411 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.117242 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" Aug 05 17:50:49.117411 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.117271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerDied","Data":"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15"} Aug 05 17:50:49.117411 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.117328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8" event={"ID":"83858385-7231-4a2d-ae1e-f4826638fa83","Type":"ContainerDied","Data":"4e52b69f056a5588f21f2c321d5aaa8dc42642b381ad6ac1faaab9159a99f073"} Aug 05 17:50:49.117411 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.117351 2564 scope.go:117] "RemoveContainer" containerID="11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15" Aug 05 17:50:49.125877 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.125858 2564 scope.go:117] "RemoveContainer" containerID="3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d" Aug 05 17:50:49.133320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.133304 2564 scope.go:117] "RemoveContainer" containerID="11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15" Aug 05 17:50:49.133560 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:50:49.133543 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15\": container with ID starting with 11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15 not found: ID does not exist" containerID="11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15" Aug 05 17:50:49.133611 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.133569 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15"} err="failed to get container status \"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15\": rpc error: code = NotFound desc = could not find container \"11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15\": container with ID starting with 11ff1909fd4ab0477370eff5d76b271f5e47fd215bcd00801d194a1530af9c15 not found: ID does not exist" Aug 05 17:50:49.133611 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.133585 2564 scope.go:117] "RemoveContainer" containerID="3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d" Aug 05 17:50:49.133818 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:50:49.133795 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d\": container with ID starting with 3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d not found: ID does not exist" containerID="3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d" Aug 05 17:50:49.133870 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.133821 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d"} err="failed to get container status \"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d\": rpc error: code = NotFound desc = could not find container \"3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d\": container with ID starting with 3442d217bdf2508e9233e163d680c16634d604fe6c4b9288c76ad5aa99c0925d not found: ID does not exist" Aug 05 17:50:49.135273 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135256 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135279 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83858385-7231-4a2d-ae1e-f4826638fa83-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135290 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135298 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk9h4\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-kube-api-access-nk9h4\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135307 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83858385-7231-4a2d-ae1e-f4826638fa83-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135315 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83858385-7231-4a2d-ae1e-f4826638fa83-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.135337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.135323 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83858385-7231-4a2d-ae1e-f4826638fa83-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:50:49.138800 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.138776 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:50:49.142562 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:49.142544 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fbbe1196-openshift-default-7b5cbdd85b-pd4f8"] Aug 05 17:50:51.039294 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:50:51.039262 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" path="/var/lib/kubelet/pods/83858385-7231-4a2d-ae1e-f4826638fa83/volumes" Aug 05 17:51:00.169702 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.169624 2564 generic.go:356] "Generic (PLEG): container finished" podID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerID="be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720" exitCode=0 Aug 05 17:51:00.169702 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.169692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerDied","Data":"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720"} Aug 05 17:51:00.170166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.170116 2564 scope.go:117] "RemoveContainer" containerID="be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720" Aug 05 17:51:00.171138 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.171109 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:00.171254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.171182 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:00.171254 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:00.171222 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:01.176014 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:01.175974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerStarted","Data":"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f"} Aug 05 17:51:01.975678 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:01.975640 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:01.976781 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:01.976752 2564 patch_prober.go:28] interesting pod/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Aug 05 17:51:01.976898 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:01.976807 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Aug 05 17:51:02.106906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:02.106865 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:02.975564 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:02.975525 2564 patch_prober.go:28] interesting pod/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Aug 05 17:51:02.975950 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:02.975582 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Aug 05 17:51:03.976150 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:03.976120 2564 patch_prober.go:28] interesting pod/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Aug 05 17:51:03.976546 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:03.976187 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Aug 05 17:51:04.976193 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:04.976155 2564 patch_prober.go:28] interesting pod/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Aug 05 17:51:04.976607 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:04.976214 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Aug 05 17:51:05.979292 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:05.979264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:05.980409 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:05.980387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:19.071323 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.071280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:19.189306 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.189270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:19.295367 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.295338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:19.376002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.375867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:19.499457 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.499428 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:51:19.499670 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.499628 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" containerID="cri-o://475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f" gracePeriod=30 Aug 05 17:51:19.527078 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.527053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:19.783342 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:19.783262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:20.002721 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.002681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:20.289865 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.289836 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:51:20.290222 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.290171 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:51:20.290222 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.290182 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:51:20.290306 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.290254 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:51:20.290306 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.290262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:51:20.292436 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.292420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.294981 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.294951 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1d0f63d4-openshift-default-dockercfg-4pkjh\"" Aug 05 17:51:20.304583 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.304563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:51:20.325998 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.325928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:20.335133 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335228 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335228 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47bw\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335379 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.335441 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.335421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436335 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47bw\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436892 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.436934 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.436883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.437061 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.437040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.437126 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.437059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.437163 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.437136 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.438745 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.438716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.439320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.439295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.446067 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.446036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.446190 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.446167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47bw\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw\") pod \"e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.528585 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.528553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:20.602673 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.602642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:20.736992 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.736903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:51:20.739301 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:51:20.739275 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb70003a_e384_47c2_ba76_27fae73de1d3.slice/crio-b8dafdfbe3f0feef2345773859f2dd488e62e26dbe8cb4fdd8b16f9db833c0c3 WatchSource:0}: Error finding container b8dafdfbe3f0feef2345773859f2dd488e62e26dbe8cb4fdd8b16f9db833c0c3: Status 404 returned error can't find the container with id b8dafdfbe3f0feef2345773859f2dd488e62e26dbe8cb4fdd8b16f9db833c0c3 Aug 05 17:51:20.741433 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.741401 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:20.741538 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.741479 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:20.741538 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.741523 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:20.981786 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.981708 2564 patch_prober.go:28] interesting pod/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:51:20.981786 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:20.981763 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:51:21.280486 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.280387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerStarted","Data":"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86"} Aug 05 17:51:21.280486 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.280447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerStarted","Data":"b8dafdfbe3f0feef2345773859f2dd488e62e26dbe8cb4fdd8b16f9db833c0c3"} Aug 05 17:51:21.301728 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.301678 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podStartSLOduration=1.301661921 podStartE2EDuration="1.301661921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:51:21.300237052 +0000 UTC m=+1914.731383302" watchObservedRunningTime="2026-08-05 17:51:21.301661921 +0000 UTC m=+1914.732808221" Aug 05 17:51:21.603586 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.603551 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:21.605120 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.605089 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:21.605269 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:21.605142 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:22.607418 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:22.607388 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:22.654792 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:22.654756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:22.752617 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:22.752586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:23.293241 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:23.293213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:23.294242 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:23.294226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:23.910360 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:23.910329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:24.147149 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.147121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:24.376514 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.376485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:24.471863 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.471833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:24.744834 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.744814 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:24.775359 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775380 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4mt\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775509 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775464 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775529 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775559 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:51:24.775668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775668 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data\") pod \"4674678e-5e8e-4f29-995a-cb985b5b562e\" (UID: \"4674678e-5e8e-4f29-995a-cb985b5b562e\") " Aug 05 17:51:24.775906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:51:24.775906 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.775866 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:51:24.776077 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776048 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:51:24.776077 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:51:24.776192 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776177 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.776247 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776197 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.776247 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776210 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.776247 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776223 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4674678e-5e8e-4f29-995a-cb985b5b562e-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.776247 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.776236 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.778117 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.778092 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:51:24.778284 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.778263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt" (OuterVolumeSpecName: "kube-api-access-lj4mt") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "kube-api-access-lj4mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:51:24.778386 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.778281 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:51:24.778386 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.778287 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4674678e-5e8e-4f29-995a-cb985b5b562e" (UID: "4674678e-5e8e-4f29-995a-cb985b5b562e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:51:24.876669 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.876643 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.876669 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.876665 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.876669 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.876673 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4674678e-5e8e-4f29-995a-cb985b5b562e-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:24.876859 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:24.876682 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4mt\" (UniqueName: \"kubernetes.io/projected/4674678e-5e8e-4f29-995a-cb985b5b562e-kube-api-access-lj4mt\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:51:25.303250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.303168 2564 generic.go:356] "Generic (PLEG): container finished" podID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerID="475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f" exitCode=0 Aug 05 17:51:25.303250 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.303214 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" Aug 05 17:51:25.303637 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.303251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerDied","Data":"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f"} Aug 05 17:51:25.303637 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.303306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4" event={"ID":"4674678e-5e8e-4f29-995a-cb985b5b562e","Type":"ContainerDied","Data":"b635cb12c30434070bd23ce44fc14c3b34720e4940d04f1598d154e61620a43e"} Aug 05 17:51:25.303637 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.303336 2564 scope.go:117] "RemoveContainer" containerID="475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f" Aug 05 17:51:25.311572 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.311552 2564 scope.go:117] "RemoveContainer" containerID="be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720" Aug 05 17:51:25.319109 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.319092 2564 scope.go:117] "RemoveContainer" containerID="475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f" Aug 05 17:51:25.319417 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:51:25.319394 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f\": container with ID starting with 475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f not found: ID does not exist" containerID="475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f" Aug 05 17:51:25.319489 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.319426 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f"} err="failed to get container status \"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f\": rpc error: code = NotFound desc = could not find container \"475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f\": container with ID starting with 475deb93f0aa3258a0dda8eb57222c83255831d390750f8ecc51f71e9df8099f not found: ID does not exist" Aug 05 17:51:25.319489 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.319444 2564 scope.go:117] "RemoveContainer" containerID="be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720" Aug 05 17:51:25.319677 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:51:25.319661 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720\": container with ID starting with be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720 not found: ID does not exist" containerID="be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720" Aug 05 17:51:25.319726 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.319679 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720"} err="failed to get container status \"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720\": rpc error: code = NotFound desc = could not find container \"be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720\": container with ID starting with be555bb96e70f4291a21b1a16113ab1fa6b7a369d7d17291df55348e70677720 not found: ID does not exist" Aug 05 17:51:25.324889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.324866 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:51:25.330680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:25.330661 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-41ad7fbb-gw-openshift-default-55744d7548-dphm4"] Aug 05 17:51:27.038077 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:27.038038 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" path="/var/lib/kubelet/pods/4674678e-5e8e-4f29-995a-cb985b5b562e/volumes" Aug 05 17:51:28.778455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:28.778388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:28.879552 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:28.879391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:36.356103 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.356075 2564 generic.go:356] "Generic (PLEG): container finished" podID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerID="4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86" exitCode=0 Aug 05 17:51:36.356475 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.356156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerDied","Data":"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86"} Aug 05 17:51:36.356577 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.356564 2564 scope.go:117] "RemoveContainer" containerID="4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86" Aug 05 17:51:36.357576 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.357550 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:36.357629 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.357618 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:36.357673 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:36.357648 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:51:37.362439 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:37.362407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerStarted","Data":"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e"} Aug 05 17:51:37.603058 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:37.603018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:37.604027 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:37.604004 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:37.604144 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:37.604049 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:38.293560 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:38.293529 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:38.603287 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:38.603207 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:38.603287 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:38.603276 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:39.603019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:39.602908 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:39.603019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:39.602987 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:40.603337 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:40.603302 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:40.603784 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:40.603370 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:41.603670 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:41.603631 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Aug 05 17:51:41.604105 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:41.603691 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Aug 05 17:51:42.606993 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:42.606946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:42.607874 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:42.607861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:51:54.292855 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:54.292811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:54.392423 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:54.392381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:54.471377 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:54.471340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:55.577517 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:55.577482 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:51:55.577914 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:55.577731 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" containerID="cri-o://2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e" gracePeriod=30 Aug 05 17:51:55.616245 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:55.616212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:55.886058 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:55.885949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:56.045626 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:56.045592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:56.381536 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:56.381496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:56.490580 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:56.490552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:51:57.609915 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:57.609877 2564 patch_prober.go:28] interesting pod/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:51:57.610332 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:51:57.609950 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:52:00.818555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.818533 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:52:00.997858 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997770 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.997858 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997803 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.997858 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997824 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997878 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v47bw\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.997933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998004 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998053 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs\") pod \"bb70003a-e384-47c2-ba76-27fae73de1d3\" (UID: \"bb70003a-e384-47c2-ba76-27fae73de1d3\") " Aug 05 17:52:00.998166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998105 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998191 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998294 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998398 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998468 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998482 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:00.998515 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:00.998492 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.000255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.000222 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:52:01.000255 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.000225 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:52:01.000415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.000313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw" (OuterVolumeSpecName: "kube-api-access-v47bw") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "kube-api-access-v47bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:52:01.000415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.000311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb70003a-e384-47c2-ba76-27fae73de1d3" (UID: "bb70003a-e384-47c2-ba76-27fae73de1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:52:01.099682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099656 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.099682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099676 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.099682 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099685 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb70003a-e384-47c2-ba76-27fae73de1d3-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.099867 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099693 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb70003a-e384-47c2-ba76-27fae73de1d3-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.099867 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099702 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47bw\" (UniqueName: \"kubernetes.io/projected/bb70003a-e384-47c2-ba76-27fae73de1d3-kube-api-access-v47bw\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.099867 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.099710 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb70003a-e384-47c2-ba76-27fae73de1d3-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:01.469258 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.469227 2564 generic.go:356] "Generic (PLEG): container finished" podID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerID="2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e" exitCode=0 Aug 05 17:52:01.469448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.469270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerDied","Data":"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e"} Aug 05 17:52:01.469448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.469283 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" Aug 05 17:52:01.469448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.469311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5" event={"ID":"bb70003a-e384-47c2-ba76-27fae73de1d3","Type":"ContainerDied","Data":"b8dafdfbe3f0feef2345773859f2dd488e62e26dbe8cb4fdd8b16f9db833c0c3"} Aug 05 17:52:01.469448 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.469333 2564 scope.go:117] "RemoveContainer" containerID="2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e" Aug 05 17:52:01.479047 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.479026 2564 scope.go:117] "RemoveContainer" containerID="4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86" Aug 05 17:52:01.487307 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.487291 2564 scope.go:117] "RemoveContainer" containerID="2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e" Aug 05 17:52:01.487573 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:52:01.487555 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e\": container with ID starting with 2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e not found: ID does not exist" containerID="2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e" Aug 05 17:52:01.487621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.487584 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e"} err="failed to get container status \"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e\": rpc error: code = NotFound desc = could not find container \"2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e\": container with ID starting with 2a8f04edb83ba9c47c066af5a4e423373b7c1d5b8ba0d0f99ddd5d3c90c44d6e not found: ID does not exist" Aug 05 17:52:01.487621 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.487600 2564 scope.go:117] "RemoveContainer" containerID="4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86" Aug 05 17:52:01.487861 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:52:01.487845 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86\": container with ID starting with 4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86 not found: ID does not exist" containerID="4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86" Aug 05 17:52:01.487910 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.487867 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86"} err="failed to get container status \"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86\": rpc error: code = NotFound desc = could not find container \"4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86\": container with ID starting with 4f596dee4001bdc2f8c3ff3e3480a79133939334b1efc76af6abd35e50cdea86 not found: ID does not exist" Aug 05 17:52:01.488429 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.488412 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:52:01.492020 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.491951 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1d0f63d4-openshift-default-54798fbb64-vdpf5"] Aug 05 17:52:01.783371 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.783250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:01.884493 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:01.884454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:02.932131 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:02.932089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:03.038559 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:03.038525 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" path="/var/lib/kubelet/pods/bb70003a-e384-47c2-ba76-27fae73de1d3/volumes" Aug 05 17:52:03.109543 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:03.109498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:03.443124 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:03.443095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:03.565860 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:03.565820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:31.776866 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.776835 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897"] Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777161 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777171 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777183 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777188 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777199 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777204 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83858385-7231-4a2d-ae1e-f4826638fa83" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777261 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:52:31.777263 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777268 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:52:31.777505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777275 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:52:31.777505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.777283 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:52:31.779019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.779005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:31.781672 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.781651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:52:31.782571 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.782555 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bz9qt\"" Aug 05 17:52:31.787577 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.787556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897"] Aug 05 17:52:31.873153 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.873124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12e4a979-4n897\" (UID: \"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:31.974170 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.974140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12e4a979-4n897\" (UID: \"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:31.974715 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:31.974698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-12e4a979-4n897\" (UID: \"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:32.089845 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:32.089822 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:32.228239 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:32.228216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897"] Aug 05 17:52:32.230028 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:52:32.230001 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397c7a9b_3522_4aa7_913e_8b8ae3a3c4b1.slice/crio-fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa WatchSource:0}: Error finding container fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa: Status 404 returned error can't find the container with id fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa Aug 05 17:52:32.621966 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:32.621932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" event={"ID":"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1","Type":"ContainerStarted","Data":"6806de29c0c1c5e59f18cdff0a8675385381db2cb779c913a6ea0656fd53d7e1"} Aug 05 17:52:32.622230 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:32.622003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" event={"ID":"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1","Type":"ContainerStarted","Data":"fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa"} Aug 05 17:52:32.638445 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:32.638395 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" podStartSLOduration=1.638380299 podStartE2EDuration="1.638380299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:52:32.636874502 +0000 UTC m=+1986.068020752" watchObservedRunningTime="2026-08-05 17:52:32.638380299 +0000 UTC m=+1986.069526549" Aug 05 17:52:33.627556 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:33.627525 2564 generic.go:356] "Generic (PLEG): container finished" podID="397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" containerID="6806de29c0c1c5e59f18cdff0a8675385381db2cb779c913a6ea0656fd53d7e1" exitCode=0 Aug 05 17:52:33.627934 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:33.627608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" event={"ID":"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1","Type":"ContainerDied","Data":"6806de29c0c1c5e59f18cdff0a8675385381db2cb779c913a6ea0656fd53d7e1"} Aug 05 17:52:34.756121 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:34.756096 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:34.901480 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:34.901410 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca\") pod \"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1\" (UID: \"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1\") " Aug 05 17:52:34.901757 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:34.901735 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" (UID: "397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:52:35.002849 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:35.002806 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1-maas-api-service-ca\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:52:35.637165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:35.637139 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" Aug 05 17:52:35.637340 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:35.637161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897" event={"ID":"397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1","Type":"ContainerDied","Data":"fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa"} Aug 05 17:52:35.637340 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:35.637217 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd114e63bf939b19ab95c4fb9f9bd0e6086c5452a3746af5114f2b6fa8977baa" Aug 05 17:52:43.473608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.473575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:43.579422 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.579389 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897"] Aug 05 17:52:43.582154 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.582129 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-12e4a979-4n897"] Aug 05 17:52:43.650240 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.650209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:43.775404 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.775331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:43.970038 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:43.970010 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:44.446019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:44.445987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:44.547746 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:44.547719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:44.648690 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:44.648656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:52:45.038498 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:45.038465 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" path="/var/lib/kubelet/pods/397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1/volumes" Aug 05 17:52:56.312588 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:52:56.312560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:02.530418 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:02.530389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:02.671088 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:02.671048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:02.782350 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:02.782273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:02.909347 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:02.909290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:03.237553 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:03.237523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:29.227547 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:29.227518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:35.455165 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:35.455113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:35.596119 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:35.596086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:35.723591 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:35.723505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:38.906572 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:38.906539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:51.569743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:51.569658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:57.386249 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:57.386216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:57.469019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:57.468987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:53:57.579434 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:53:57.579405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:08.168279 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:08.168244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:19.204112 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:19.204077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:25.078495 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:25.078466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:25.174838 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:25.174808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:25.282264 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:25.282225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:25.379124 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:25.379041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:26.021661 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:26.021626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:26.120656 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:26.120621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:26.219576 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:26.219548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:27.787075 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:27.787034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:34.421661 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:34.421626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:34.526173 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:34.526143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:35.149031 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:35.148996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:41.715689 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:41.715658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:41.815404 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:41.815366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:42.011317 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.011240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:42.253512 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253484 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:54:42.253853 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253841 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:54:42.253900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253856 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb70003a-e384-47c2-ba76-27fae73de1d3" containerName="istio-proxy" Aug 05 17:54:42.253900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253868 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:54:42.253900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253874 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4674678e-5e8e-4f29-995a-cb985b5b562e" containerName="istio-proxy" Aug 05 17:54:42.253900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253880 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" containerName="revoke-keys" Aug 05 17:54:42.253900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253886 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" containerName="revoke-keys" Aug 05 17:54:42.254107 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.253931 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="397c7a9b-3522-4aa7-913e-8b8ae3a3c4b1" containerName="revoke-keys" Aug 05 17:54:42.255642 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.255628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.258275 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.258252 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4f48a67f-openshift-default-dockercfg-z7spd\"" Aug 05 17:54:42.275563 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.275539 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:54:42.327522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327701 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fg8\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.327889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.327829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.418447 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.418413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:42.428745 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fg8\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.428873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.428873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.428873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.428873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.428873 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.428914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429166 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429318 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429318 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429422 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429638 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.429727 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.429692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.431439 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.431416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.431723 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.431701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.436916 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.436886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.437027 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.436918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fg8\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8\") pod \"e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.565555 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.565513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:42.702522 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.702494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:42.716843 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.716821 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:54:42.719194 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:54:42.719166 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd5c873_7937_454a_a364_b78b40c461e0.slice/crio-8f9f5f403e260a0227992b63d1529fc755bd14e0a4e103f14fc6a81ca34e57ad WatchSource:0}: Error finding container 8f9f5f403e260a0227992b63d1529fc755bd14e0a4e103f14fc6a81ca34e57ad: Status 404 returned error can't find the container with id 8f9f5f403e260a0227992b63d1529fc755bd14e0a4e103f14fc6a81ca34e57ad Aug 05 17:54:42.721455 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.721423 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:54:42.721536 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.721488 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:54:42.721573 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:42.721534 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:54:43.197147 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.197112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerStarted","Data":"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156"} Aug 05 17:54:43.197320 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.197175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerStarted","Data":"8f9f5f403e260a0227992b63d1529fc755bd14e0a4e103f14fc6a81ca34e57ad"} Aug 05 17:54:43.217808 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.217754 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podStartSLOduration=1.217737521 podStartE2EDuration="1.217737521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:54:43.216538686 +0000 UTC m=+2116.647684937" watchObservedRunningTime="2026-08-05 17:54:43.217737521 +0000 UTC m=+2116.648883773" Aug 05 17:54:43.566192 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.566162 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:43.567665 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.567637 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:54:43.567721 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:43.567683 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:54:44.570077 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:44.570050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:45.165883 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:45.165845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:45.207918 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:45.207885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:45.208913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:45.208892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:54:45.265743 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:45.265706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:46.298617 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:46.298589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:46.401856 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:46.401828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:46.831986 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:46.831930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:46.936068 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:46.936026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:54:59.272900 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.272860 2564 generic.go:356] "Generic (PLEG): container finished" podID="2bd5c873-7937-454a-a364-b78b40c461e0" containerID="b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156" exitCode=0 Aug 05 17:54:59.273348 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.272939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerDied","Data":"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156"} Aug 05 17:54:59.273462 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.273442 2564 scope.go:117] "RemoveContainer" containerID="b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156" Aug 05 17:54:59.274519 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.274490 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:54:59.274572 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.274564 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:54:59.274635 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:54:59.274623 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:55:00.208052 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:00.208010 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:00.279674 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:00.279632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerStarted","Data":"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc"} Aug 05 17:55:00.566297 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:00.566266 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:00.567384 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:00.567357 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:55:00.567505 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:00.567409 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:55:01.566034 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:01.565996 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:55:01.566426 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:01.566045 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:55:02.566540 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:02.566500 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:55:02.566937 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:02.566559 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:55:03.566745 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:03.566706 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:55:03.567218 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:03.566765 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:55:04.566712 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:04.566674 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Aug 05 17:55:04.566905 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:04.566730 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Aug 05 17:55:05.569889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:05.569862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:05.570324 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:05.570133 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:05.570788 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:05.570774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:10.253353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:10.253317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:10.919090 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:10.919060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:11.061993 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:11.061945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:11.396639 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:11.396613 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:17.033139 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:17.033106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:17.103992 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:17.103947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:27.898754 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:27.898678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:28.010353 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:28.010320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:28.121762 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:28.121726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:28.353116 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:28.353085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:28.499343 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:28.499314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.228019 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.227986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.373849 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.373817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.434805 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.434771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:55:34.435329 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.435296 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" containerID="cri-o://c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc" gracePeriod=30 Aug 05 17:55:34.545528 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.545475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.705098 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.705062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.843634 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.843535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:34.933762 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:34.933726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:35.055929 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:35.055893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:35.572985 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:35.572935 2564 patch_prober.go:28] interesting pod/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:55:35.573414 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:35.573043 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:55:39.677821 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.677801 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:39.731773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731704 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.731773 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731756 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731884 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731939 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732002 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.731994 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4fg8\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732034 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo\") pod \"2bd5c873-7937-454a-a364-b78b40c461e0\" (UID: \"2bd5c873-7937-454a-a364-b78b40c461e0\") " Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732229 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:55:39.732396 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732302 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:55:39.732909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732441 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.732909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732467 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.732909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732478 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.732909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732490 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.732909 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.732505 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd5c873-7937-454a-a364-b78b40c461e0-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.734130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.734106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:55:39.734323 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.734296 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:55:39.734323 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.734308 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:55:39.734323 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.734316 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8" (OuterVolumeSpecName: "kube-api-access-j4fg8") pod "2bd5c873-7937-454a-a364-b78b40c461e0" (UID: "2bd5c873-7937-454a-a364-b78b40c461e0"). InnerVolumeSpecName "kube-api-access-j4fg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:55:39.833878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.833843 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4fg8\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-kube-api-access-j4fg8\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.833878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.833878 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd5c873-7937-454a-a364-b78b40c461e0-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.833878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.833888 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd5c873-7937-454a-a364-b78b40c461e0-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:39.834133 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:39.833896 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd5c873-7937-454a-a364-b78b40c461e0-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 17:55:40.463197 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.463160 2564 generic.go:356] "Generic (PLEG): container finished" podID="2bd5c873-7937-454a-a364-b78b40c461e0" containerID="c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc" exitCode=0 Aug 05 17:55:40.463351 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.463215 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" Aug 05 17:55:40.463351 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.463227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerDied","Data":"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc"} Aug 05 17:55:40.463351 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.463281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85" event={"ID":"2bd5c873-7937-454a-a364-b78b40c461e0","Type":"ContainerDied","Data":"8f9f5f403e260a0227992b63d1529fc755bd14e0a4e103f14fc6a81ca34e57ad"} Aug 05 17:55:40.463351 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.463309 2564 scope.go:117] "RemoveContainer" containerID="c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc" Aug 05 17:55:40.472004 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.471985 2564 scope.go:117] "RemoveContainer" containerID="b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156" Aug 05 17:55:40.479728 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.479713 2564 scope.go:117] "RemoveContainer" containerID="c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc" Aug 05 17:55:40.479998 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:55:40.479980 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc\": container with ID starting with c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc not found: ID does not exist" containerID="c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc" Aug 05 17:55:40.480054 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.480008 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc"} err="failed to get container status \"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc\": rpc error: code = NotFound desc = could not find container \"c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc\": container with ID starting with c02884613291b2c0f1392d79dcf257ff8a92d351a739e0e4497eb77d552641cc not found: ID does not exist" Aug 05 17:55:40.480054 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.480027 2564 scope.go:117] "RemoveContainer" containerID="b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156" Aug 05 17:55:40.480281 ip-10-0-138-228 kubenswrapper[2564]: E0805 17:55:40.480264 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156\": container with ID starting with b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156 not found: ID does not exist" containerID="b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156" Aug 05 17:55:40.480331 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.480290 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156"} err="failed to get container status \"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156\": rpc error: code = NotFound desc = could not find container \"b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156\": container with ID starting with b5bc0914624564aef0943687b979c42ace95a821f7331a442f39122ff02b4156 not found: ID does not exist" Aug 05 17:55:40.484230 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.484209 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:55:40.488598 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.488578 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4f48a67f-openshift-default-684c75cb88-lbc85"] Aug 05 17:55:40.797674 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:40.797648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:41.039201 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:41.039168 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" path="/var/lib/kubelet/pods/2bd5c873-7937-454a-a364-b78b40c461e0/volumes" Aug 05 17:55:42.942966 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:42.942924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:43.692358 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:43.692325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:46.767700 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:46.767666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:55:46.917116 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:55:46.917080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:36.523595 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:36.523556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:49.650592 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.650560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:49.915612 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915535 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 17:57:49.915874 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915862 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.915874 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915875 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.916009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915886 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.916009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915893 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.916009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915947 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.916009 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.915969 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd5c873-7937-454a-a364-b78b40c461e0" containerName="istio-proxy" Aug 05 17:57:49.918937 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.918920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:49.921413 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.921392 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b60f38ad-openshift-default-dockercfg-ngmmt\"" Aug 05 17:57:49.928680 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.928661 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 17:57:49.940549 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:49.940523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:50.071751 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gq5\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.071922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.071922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.071922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.071922 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.072104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.071974 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.072104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.072022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.072104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.072048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.072104 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.072062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173196 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173196 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173196 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gq5\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173451 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173744 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.173854 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.173785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.174368 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.174344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.175656 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.175630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.175845 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.175831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.182913 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.182859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.183168 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.183151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gq5\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5\") pod \"e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.228896 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.228876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:50.359674 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.359652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 17:57:50.361738 ip-10-0-138-228 kubenswrapper[2564]: W0805 17:57:50.361713 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fbe4f6_4824_4475_af24_a3abeb393ab5.slice/crio-cc48dc1c18f0ed1340f10ece266657f8379a8aaaa53d954b941870d71f757b1c WatchSource:0}: Error finding container cc48dc1c18f0ed1340f10ece266657f8379a8aaaa53d954b941870d71f757b1c: Status 404 returned error can't find the container with id cc48dc1c18f0ed1340f10ece266657f8379a8aaaa53d954b941870d71f757b1c Aug 05 17:57:50.363901 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.363867 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:57:50.363997 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.363950 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:57:50.364043 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.364013 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:57:50.464249 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:50.464221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:51.032614 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:51.032581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerStarted","Data":"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1"} Aug 05 17:57:51.038415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:51.038393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerStarted","Data":"cc48dc1c18f0ed1340f10ece266657f8379a8aaaa53d954b941870d71f757b1c"} Aug 05 17:57:51.054056 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:51.054015 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podStartSLOduration=2.054002221 podStartE2EDuration="2.054002221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:57:51.051786665 +0000 UTC m=+2304.482932916" watchObservedRunningTime="2026-08-05 17:57:51.054002221 +0000 UTC m=+2304.485148470" Aug 05 17:57:51.229641 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:51.229612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:51.234344 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:51.234324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:52.037910 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:52.037882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:52.038889 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:52.038873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:57:52.964149 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:52.964117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:53.067194 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:53.067160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:54.110301 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:54.110273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:54.254189 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:54.254157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:54.639252 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:54.639219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:57:54.741435 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:57:54.741400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:07.570874 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:07.570837 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:32792->10.133.0.38:15021: read: connection reset by peer" start-of-body= Aug 05 17:58:07.571411 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:07.570912 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:32792->10.133.0.38:15021: read: connection reset by peer" Aug 05 17:58:08.116878 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.116844 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerID="6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1" exitCode=0 Aug 05 17:58:08.117111 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.116921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerDied","Data":"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1"} Aug 05 17:58:08.117389 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.117373 2564 scope.go:117] "RemoveContainer" containerID="6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1" Aug 05 17:58:08.118381 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.118353 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:58:08.118470 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.118437 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:58:08.118530 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:08.118486 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:58:09.122841 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:09.122805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerStarted","Data":"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9"} Aug 05 17:58:09.229137 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:09.229101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:58:09.229328 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:09.229304 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 17:58:09.229393 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:09.229374 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 17:58:10.229837 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:10.229802 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 17:58:10.230256 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:10.229856 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 17:58:11.230149 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:11.230116 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 17:58:11.230608 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:11.230179 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 17:58:12.230040 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:12.230005 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 17:58:12.230218 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:12.230062 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 17:58:13.229809 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:13.229775 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 17:58:13.230013 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:13.229837 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 17:58:14.232599 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:14.232571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:58:14.232991 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:14.232824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:58:14.233415 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:14.233400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 17:58:17.963467 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:17.963388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:19.236437 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:19.236404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:29.548277 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:29.548244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:37.851591 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:37.851560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:50.685324 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:50.685292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:58:51.310200 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:58:51.310170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:18.586972 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:18.586914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:18.684582 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:18.684545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:18.764064 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:18.764034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:22.033550 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:22.033519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:27.067091 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:27.067059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:27.175857 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:27.175828 2564 scope.go:117] "RemoveContainer" containerID="6806de29c0c1c5e59f18cdff0a8675385381db2cb779c913a6ea0656fd53d7e1" Aug 05 17:59:44.065168 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:44.065126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 17:59:44.166130 ip-10-0-138-228 kubenswrapper[2564]: I0805 17:59:44.166093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:03.629124 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:03.629092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:04.255140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:04.255100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:09.618728 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:09.618696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:09.722382 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:09.722329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:09.877675 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:09.877599 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 18:00:09.877823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:09.877798 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" containerID="cri-o://83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9" gracePeriod=30 Aug 05 18:00:09.902015 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:09.901986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:10.138779 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:10.138690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:14.234971 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:14.234930 2564 patch_prober.go:28] interesting pod/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:00:14.235336 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:14.235014 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:00:15.127549 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.127524 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 18:00:15.194519 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194435 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194519 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194483 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194519 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9gq5\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194613 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194685 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194713 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.194806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket\") pod \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\" (UID: \"b9fbe4f6-4824-4475-af24-a3abeb393ab5\") " Aug 05 18:00:15.195123 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:00:15.195123 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.194850 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:00:15.195123 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.195093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:00:15.195249 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.195142 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.195249 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.195180 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.195374 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.195347 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:00:15.195490 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.195465 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:00:15.196739 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.196710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:00:15.196884 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.196839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:00:15.197188 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.197158 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:00:15.197319 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.197300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5" (OuterVolumeSpecName: "kube-api-access-p9gq5") pod "b9fbe4f6-4824-4475-af24-a3abeb393ab5" (UID: "b9fbe4f6-4824-4475-af24-a3abeb393ab5"). InnerVolumeSpecName "kube-api-access-p9gq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:00:15.296426 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296399 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296426 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296423 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296426 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296432 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296446 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296455 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296462 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9gq5\" (UniqueName: \"kubernetes.io/projected/b9fbe4f6-4824-4475-af24-a3abeb393ab5-kube-api-access-p9gq5\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.296823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.296469 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fbe4f6-4824-4475-af24-a3abeb393ab5-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:00:15.688232 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.688195 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerID="83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9" exitCode=0 Aug 05 18:00:15.688403 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.688245 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" Aug 05 18:00:15.688403 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.688277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerDied","Data":"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9"} Aug 05 18:00:15.688403 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.688335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx" event={"ID":"b9fbe4f6-4824-4475-af24-a3abeb393ab5","Type":"ContainerDied","Data":"cc48dc1c18f0ed1340f10ece266657f8379a8aaaa53d954b941870d71f757b1c"} Aug 05 18:00:15.688403 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.688356 2564 scope.go:117] "RemoveContainer" containerID="83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9" Aug 05 18:00:15.702593 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.697785 2564 scope.go:117] "RemoveContainer" containerID="6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1" Aug 05 18:00:15.706616 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.706600 2564 scope.go:117] "RemoveContainer" containerID="83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9" Aug 05 18:00:15.706862 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:00:15.706845 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9\": container with ID starting with 83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9 not found: ID does not exist" containerID="83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9" Aug 05 18:00:15.706908 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.706870 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9"} err="failed to get container status \"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9\": rpc error: code = NotFound desc = could not find container \"83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9\": container with ID starting with 83ad0ccc1ef7c7f42d8b4cbf76f31bd36ec753a763ee0172b070efef5a214aa9 not found: ID does not exist" Aug 05 18:00:15.706908 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.706887 2564 scope.go:117] "RemoveContainer" containerID="6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1" Aug 05 18:00:15.707172 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:00:15.707156 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1\": container with ID starting with 6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1 not found: ID does not exist" containerID="6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1" Aug 05 18:00:15.707219 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.707177 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1"} err="failed to get container status \"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1\": rpc error: code = NotFound desc = could not find container \"6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1\": container with ID starting with 6ad81708e1ea7741371d75b9a5ae52f579c8c2de93b933758593d1fd3f32b8f1 not found: ID does not exist" Aug 05 18:00:15.716157 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.716126 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 18:00:15.722307 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:15.722286 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b60f38ad-openshift-default-b486c768d-cpjcx"] Aug 05 18:00:17.039363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:17.039326 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" path="/var/lib/kubelet/pods/b9fbe4f6-4824-4475-af24-a3abeb393ab5/volumes" Aug 05 18:00:21.584766 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:21.584735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:21.686069 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:21.686038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:22.304107 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:22.304072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:22.404148 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:22.404115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:22.502617 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:22.502583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:23.846063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:23.846031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:30.000678 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:30.000644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:30.105668 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:30.105633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:30.680292 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:30.680262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:37.662864 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:37.662826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:37.763101 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:37.763068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:37.862749 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:37.862721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:48.356835 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:48.356798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:00:50.667100 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:00:50.667061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:09.612226 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:09.612195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:15.476637 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:15.476600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:15.555003 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:15.554975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:15.916109 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:15.916078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:16.103936 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.103906 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:01:16.104408 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104384 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.104408 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104407 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.104594 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104423 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.104594 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104439 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.104594 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104542 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.104594 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.104553 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fbe4f6-4824-4475-af24-a3abeb393ab5" containerName="istio-proxy" Aug 05 18:01:16.107237 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.107217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.109854 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.109832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-19f4b3-openshift-default-dockercfg-pwk5c\"" Aug 05 18:01:16.118210 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.118190 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:01:16.193938 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.193852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:16.246113 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246113 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246233 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7p6g\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246536 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.246536 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.246414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.315735 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.315707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:16.347910 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.347884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.347923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.347941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7p6g\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348285 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348285 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348390 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348455 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348455 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348579 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348579 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348696 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.348757 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.348704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.350306 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.350291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.350805 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.350785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.357941 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.357916 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.358269 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.358243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7p6g\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g\") pod \"e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.417359 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.417335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:16.549587 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.549549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:01:16.551950 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:01:16.551926 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f0657_e32b_4b6b_802b_426367859f87.slice/crio-a1fc3ea2d5e0a8e7ca0b1b7ab6cf75fd04a4e7f9081b41b8cb18fec84be89054 WatchSource:0}: Error finding container a1fc3ea2d5e0a8e7ca0b1b7ab6cf75fd04a4e7f9081b41b8cb18fec84be89054: Status 404 returned error can't find the container with id a1fc3ea2d5e0a8e7ca0b1b7ab6cf75fd04a4e7f9081b41b8cb18fec84be89054 Aug 05 18:01:16.554593 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.554560 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:16.554681 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.554635 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:16.554681 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.554674 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:16.948005 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.947950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerStarted","Data":"acd9d7b46e8c4960c65f48d9dbe176d495de53131835f15b8c7825756cb77ef4"} Aug 05 18:01:16.948180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.948020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerStarted","Data":"a1fc3ea2d5e0a8e7ca0b1b7ab6cf75fd04a4e7f9081b41b8cb18fec84be89054"} Aug 05 18:01:16.968484 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:16.968440 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podStartSLOduration=0.968419182 podStartE2EDuration="968.419182ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:01:16.968248956 +0000 UTC m=+2510.399395206" watchObservedRunningTime="2026-08-05 18:01:16.968419182 +0000 UTC m=+2510.399565431" Aug 05 18:01:17.417719 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:17.417688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:17.423636 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:17.423611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:17.952725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:17.952695 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:17.953683 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:17.953665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:18.949518 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:18.949484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:19.052149 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:19.052115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:20.260695 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:20.260664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:20.350915 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:20.350828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:20.809423 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:20.809395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:20.893654 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:20.893623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:32.953911 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:32.953882 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:32.954267 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:32.953930 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:33.026420 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.026360 2564 generic.go:356] "Generic (PLEG): container finished" podID="c03f0657-e32b-4b6b-802b-426367859f87" containerID="acd9d7b46e8c4960c65f48d9dbe176d495de53131835f15b8c7825756cb77ef4" exitCode=0 Aug 05 18:01:33.026420 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.026390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerDied","Data":"acd9d7b46e8c4960c65f48d9dbe176d495de53131835f15b8c7825756cb77ef4"} Aug 05 18:01:33.026745 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.026730 2564 scope.go:117] "RemoveContainer" containerID="acd9d7b46e8c4960c65f48d9dbe176d495de53131835f15b8c7825756cb77ef4" Aug 05 18:01:33.027736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.027704 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:33.027825 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.027793 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:33.027865 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:33.027830 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:01:34.033063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:34.033022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerStarted","Data":"b9436f56ca759402a51bb2755a9054b9bd6ba3683102c8f64d7480438e9c45c4"} Aug 05 18:01:34.418222 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:34.418191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:34.418397 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:34.418324 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:34.418397 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:34.418362 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:35.417732 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:35.417698 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:35.418110 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:35.417756 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:36.418142 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:36.418107 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:36.418536 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:36.418174 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:37.418198 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:37.418162 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:37.418570 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:37.418227 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:37.735686 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:37.735609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:38.417742 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:38.417706 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:01:38.417917 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:38.417762 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Aug 05 18:01:39.421610 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:39.421583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:39.422016 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:39.421861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:39.422391 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:39.422377 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:01:43.925199 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:43.925165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:44.272387 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:44.272310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:44.583364 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:44.583332 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:44.905287 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:44.905208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:48.359617 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:48.359587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:48.471234 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:48.471201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:48.803461 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:48.803430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:49.017662 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:49.017629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:49.309420 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:49.309380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:01:49.446780 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:01:49.446753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:02:12.732244 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:12.732204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:02:17.500195 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:17.500157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:02:51.762000 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.761893 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:02:51.764792 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.764770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.767382 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.767359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76d454\"/\"default-dockercfg-ssdsd\"" Aug 05 18:02:51.767492 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.767359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76d454\"/\"kube-root-ca.crt\"" Aug 05 18:02:51.768513 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.768491 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76d454\"/\"openshift-service-ca.crt\"" Aug 05 18:02:51.768620 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.768605 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76d454\"/\"test-model-76d454-kserve-self-signed-certs\"" Aug 05 18:02:51.775607 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.775589 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:02:51.796991 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.796946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:02:51.812286 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.812469 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.812469 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.812469 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nx2\" (UniqueName: \"kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.812469 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.812469 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.812408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913121 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nx2\" (UniqueName: \"kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913660 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913660 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.913660 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.913645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.915621 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.915597 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.915899 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.915880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:51.921524 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:51.921505 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nx2\" (UniqueName: \"kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2\") pod \"test-model-76d454-kserve-577977646f-gb7fg\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:52.088729 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:52.088690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:02:52.216198 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:52.216175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:02:52.218219 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:02:52.218195 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf067a8a6_c53d_49fd_93a7_2f276e9f55ab.slice/crio-1cb69f010f81cece329425041adee08faca171f96d3ac88cbd74b0e162a024dd WatchSource:0}: Error finding container 1cb69f010f81cece329425041adee08faca171f96d3ac88cbd74b0e162a024dd: Status 404 returned error can't find the container with id 1cb69f010f81cece329425041adee08faca171f96d3ac88cbd74b0e162a024dd Aug 05 18:02:52.365449 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:52.365369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" event={"ID":"f067a8a6-c53d-49fd-93a7-2f276e9f55ab","Type":"ContainerStarted","Data":"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe"} Aug 05 18:02:52.365449 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:52.365411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" event={"ID":"f067a8a6-c53d-49fd-93a7-2f276e9f55ab","Type":"ContainerStarted","Data":"1cb69f010f81cece329425041adee08faca171f96d3ac88cbd74b0e162a024dd"} Aug 05 18:02:52.383730 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:52.383680 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" podStartSLOduration=1.38366331 podStartE2EDuration="1.38366331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:02:52.381353137 +0000 UTC m=+2605.812499387" watchObservedRunningTime="2026-08-05 18:02:52.38366331 +0000 UTC m=+2605.814809560" Aug 05 18:02:54.106212 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:02:54.106150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:02.088810 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:02.088772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:02.088810 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:02.088821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:02.100911 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:02.100891 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:02.423500 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:02.423422 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:32.327053 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:32.327009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:33.245103 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:33.245072 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:03:33.329915 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:33.329886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:36.415214 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:36.415165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:36.637555 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:36.637523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:43.873908 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:43.873871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:43.991652 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:43.991612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:44.125587 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:44.125504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:44.406917 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:44.406834 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:48.264347 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.264287 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" podUID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" containerName="main" containerID="cri-o://a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe" gracePeriod=45 Aug 05 18:03:48.512341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.512317 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:48.618969 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.618914 2564 generic.go:356] "Generic (PLEG): container finished" podID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" containerID="a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe" exitCode=0 Aug 05 18:03:48.619174 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.619001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" event={"ID":"f067a8a6-c53d-49fd-93a7-2f276e9f55ab","Type":"ContainerDied","Data":"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe"} Aug 05 18:03:48.619174 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.619056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" event={"ID":"f067a8a6-c53d-49fd-93a7-2f276e9f55ab","Type":"ContainerDied","Data":"1cb69f010f81cece329425041adee08faca171f96d3ac88cbd74b0e162a024dd"} Aug 05 18:03:48.619174 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.619078 2564 scope.go:117] "RemoveContainer" containerID="a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe" Aug 05 18:03:48.619353 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.619287 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg" Aug 05 18:03:48.627897 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.627865 2564 scope.go:117] "RemoveContainer" containerID="a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe" Aug 05 18:03:48.628263 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:03:48.628244 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe\": container with ID starting with a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe not found: ID does not exist" containerID="a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe" Aug 05 18:03:48.628321 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.628275 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe"} err="failed to get container status \"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe\": rpc error: code = NotFound desc = could not find container \"a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe\": container with ID starting with a56ead74090c99daa0bd3597e6bedcf320ae2831c9eb8ace5b0321a6f95c02fe not found: ID does not exist" Aug 05 18:03:48.661723 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.661723 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.661916 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.661916 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661792 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.661916 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.661916 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661885 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7nx2\" (UniqueName: \"kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2\") pod \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\" (UID: \"f067a8a6-c53d-49fd-93a7-2f276e9f55ab\") " Aug 05 18:03:48.662120 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.661992 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:48.662185 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.662155 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:48.662243 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.662173 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home" (OuterVolumeSpecName: "home") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:48.662299 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.662272 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-home\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.662299 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.662287 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-model-cache\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.662299 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.662297 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tmp-dir\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.664236 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.664206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2" (OuterVolumeSpecName: "kube-api-access-f7nx2") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "kube-api-access-f7nx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:03:48.664236 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.664217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm" (OuterVolumeSpecName: "dshm") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:48.664236 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.664224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f067a8a6-c53d-49fd-93a7-2f276e9f55ab" (UID: "f067a8a6-c53d-49fd-93a7-2f276e9f55ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:03:48.763566 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.763528 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nx2\" (UniqueName: \"kubernetes.io/projected/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-kube-api-access-f7nx2\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.763566 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.763560 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-tls-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.763566 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.763570 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f067a8a6-c53d-49fd-93a7-2f276e9f55ab-dshm\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:48.939535 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.939501 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:03:48.942810 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:48.942786 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-76d454/test-model-76d454-kserve-577977646f-gb7fg"] Aug 05 18:03:49.039744 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:49.039713 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" path="/var/lib/kubelet/pods/f067a8a6-c53d-49fd-93a7-2f276e9f55ab/volumes" Aug 05 18:03:50.290374 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.290341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:50.392206 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.392162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:50.482929 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.482896 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:03:50.483202 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.483170 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" containerID="cri-o://b9436f56ca759402a51bb2755a9054b9bd6ba3683102c8f64d7480438e9c45c4" gracePeriod=30 Aug 05 18:03:50.660303 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.660257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:50.758472 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.758438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:03:50.853423 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.853394 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5"] Aug 05 18:03:50.853770 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.853757 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" containerName="main" Aug 05 18:03:50.853812 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.853772 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" containerName="main" Aug 05 18:03:50.853846 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.853839 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f067a8a6-c53d-49fd-93a7-2f276e9f55ab" containerName="main" Aug 05 18:03:50.856732 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.856718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:50.859129 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.859107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:03:50.859389 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.859367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bz9qt\"" Aug 05 18:03:50.863621 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.863599 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5"] Aug 05 18:03:50.985297 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:50.985191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76d454-28xt5\" (UID: \"6735c477-206b-47f4-b94c-3b53d25e1240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:51.086160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.086125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76d454-28xt5\" (UID: \"6735c477-206b-47f4-b94c-3b53d25e1240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:51.086738 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.086719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-76d454-28xt5\" (UID: \"6735c477-206b-47f4-b94c-3b53d25e1240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:51.167260 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.167233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:51.295473 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.295450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5"] Aug 05 18:03:51.297903 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:03:51.297876 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6735c477_206b_47f4_b94c_3b53d25e1240.slice/crio-1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa WatchSource:0}: Error finding container 1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa: Status 404 returned error can't find the container with id 1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa Aug 05 18:03:51.637137 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.637103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" event={"ID":"6735c477-206b-47f4-b94c-3b53d25e1240","Type":"ContainerStarted","Data":"5469c2ed57c6a203a9dff5d450bd006ba65f6ee740506b1426addef246d02525"} Aug 05 18:03:51.637338 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.637166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" event={"ID":"6735c477-206b-47f4-b94c-3b53d25e1240","Type":"ContainerStarted","Data":"1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa"} Aug 05 18:03:51.655220 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:51.655172 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" podStartSLOduration=1.655156908 podStartE2EDuration="1.655156908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:03:51.65340168 +0000 UTC m=+2665.084547930" watchObservedRunningTime="2026-08-05 18:03:51.655156908 +0000 UTC m=+2665.086303158" Aug 05 18:03:52.642745 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:52.642711 2564 generic.go:356] "Generic (PLEG): container finished" podID="6735c477-206b-47f4-b94c-3b53d25e1240" containerID="5469c2ed57c6a203a9dff5d450bd006ba65f6ee740506b1426addef246d02525" exitCode=0 Aug 05 18:03:52.643201 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:52.642791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" event={"ID":"6735c477-206b-47f4-b94c-3b53d25e1240","Type":"ContainerDied","Data":"5469c2ed57c6a203a9dff5d450bd006ba65f6ee740506b1426addef246d02525"} Aug 05 18:03:53.772300 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:53.772276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:53.913401 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:53.913313 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca\") pod \"6735c477-206b-47f4-b94c-3b53d25e1240\" (UID: \"6735c477-206b-47f4-b94c-3b53d25e1240\") " Aug 05 18:03:53.913661 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:53.913638 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6735c477-206b-47f4-b94c-3b53d25e1240" (UID: "6735c477-206b-47f4-b94c-3b53d25e1240"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:03:54.014266 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.014223 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6735c477-206b-47f4-b94c-3b53d25e1240-maas-api-service-ca\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:54.424611 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.424578 2564 patch_prober.go:28] interesting pod/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:03:54.424791 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.424636 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:03:54.655549 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.655517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" event={"ID":"6735c477-206b-47f4-b94c-3b53d25e1240","Type":"ContainerDied","Data":"1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa"} Aug 05 18:03:54.655549 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.655548 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5" Aug 05 18:03:54.655748 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:54.655563 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd9fe76d1b60744246d69a852c05a91c67bba04b1f43c65667ae3fbe4d3b7fa" Aug 05 18:03:55.663037 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.663002 2564 generic.go:356] "Generic (PLEG): container finished" podID="c03f0657-e32b-4b6b-802b-426367859f87" containerID="b9436f56ca759402a51bb2755a9054b9bd6ba3683102c8f64d7480438e9c45c4" exitCode=0 Aug 05 18:03:55.663464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.663049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerDied","Data":"b9436f56ca759402a51bb2755a9054b9bd6ba3683102c8f64d7480438e9c45c4"} Aug 05 18:03:55.663464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.663122 2564 scope.go:117] "RemoveContainer" containerID="acd9d7b46e8c4960c65f48d9dbe176d495de53131835f15b8c7825756cb77ef4" Aug 05 18:03:55.730965 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.730928 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:03:55.829608 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.829797 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.829797 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.829797 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.829797 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829706 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.829797 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7p6g\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.830076 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.830076 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.830076 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert\") pod \"c03f0657-e32b-4b6b-802b-426367859f87\" (UID: \"c03f0657-e32b-4b6b-802b-426367859f87\") " Aug 05 18:03:55.830076 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.829993 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data" (OuterVolumeSpecName: "istio-data") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:55.830076 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:55.830333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:55.830333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830287 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.830333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830309 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.830476 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830411 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:03:55.830563 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.830522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:55.832087 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.832060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:03:55.832671 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.832653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token" (OuterVolumeSpecName: "istio-token") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:03:55.832767 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.832721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:03:55.832826 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.832765 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g" (OuterVolumeSpecName: "kube-api-access-j7p6g") pod "c03f0657-e32b-4b6b-802b-426367859f87" (UID: "c03f0657-e32b-4b6b-802b-426367859f87"). InnerVolumeSpecName "kube-api-access-j7p6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:03:55.931258 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931213 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f0657-e32b-4b6b-802b-426367859f87-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931258 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931248 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931258 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931261 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931512 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931273 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7p6g\" (UniqueName: \"kubernetes.io/projected/c03f0657-e32b-4b6b-802b-426367859f87-kube-api-access-j7p6g\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931512 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931286 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931512 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931297 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f0657-e32b-4b6b-802b-426367859f87-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:55.931512 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:55.931308 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f0657-e32b-4b6b-802b-426367859f87-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:03:56.669537 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:56.669511 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" Aug 05 18:03:56.669537 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:56.669523 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh" event={"ID":"c03f0657-e32b-4b6b-802b-426367859f87","Type":"ContainerDied","Data":"a1fc3ea2d5e0a8e7ca0b1b7ab6cf75fd04a4e7f9081b41b8cb18fec84be89054"} Aug 05 18:03:56.670073 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:56.669577 2564 scope.go:117] "RemoveContainer" containerID="b9436f56ca759402a51bb2755a9054b9bd6ba3683102c8f64d7480438e9c45c4" Aug 05 18:03:56.691257 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:56.691229 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:03:56.695289 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:56.695266 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-19f4b3-openshift-default-59fcf75688-bf8nh"] Aug 05 18:03:57.039306 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:57.039230 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03f0657-e32b-4b6b-802b-426367859f87" path="/var/lib/kubelet/pods/c03f0657-e32b-4b6b-802b-426367859f87/volumes" Aug 05 18:03:59.554513 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:03:59.554476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:01.321512 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.321466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:01.429350 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.429318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:01.463249 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.463214 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5"] Aug 05 18:04:01.465553 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.465529 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-76d454-28xt5"] Aug 05 18:04:01.531209 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.531176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:01.864542 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:01.864500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:03.038383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:03.038348 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6735c477-206b-47f4-b94c-3b53d25e1240" path="/var/lib/kubelet/pods/6735c477-206b-47f4-b94c-3b53d25e1240/volumes" Aug 05 18:04:07.836204 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:07.836167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:07.962810 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:07.962771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:08.131163 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.131080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:08.312587 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.312554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:08.664274 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.664242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:08.951673 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.951593 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.951913 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.951923 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.951930 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6735c477-206b-47f4-b94c-3b53d25e1240" containerName="revoke-keys" Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.951935 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6735c477-206b-47f4-b94c-3b53d25e1240" containerName="revoke-keys" Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.952025 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:08.952052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.952034 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6735c477-206b-47f4-b94c-3b53d25e1240" containerName="revoke-keys" Aug 05 18:04:08.954058 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.954040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:08.956228 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.956207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d5bdd1-openshift-default-dockercfg-rhswv\"" Aug 05 18:04:08.964981 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.964938 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:04:08.975042 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:08.975014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:09.049851 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.049808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.050116 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.050096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.050301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.050274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.050448 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.050426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jqx\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.050513 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.050498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.050571 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.050558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.051034 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.051012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.051127 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.051111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.051183 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.051169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152102 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152102 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jqx\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152363 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152718 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152718 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152718 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.152899 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.152817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.153126 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.153109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.154631 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.154610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.154745 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.154728 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.161005 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.160981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.161088 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.160987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jqx\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx\") pod \"e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.195346 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.195300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:09.265411 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.265327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:09.397556 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.397523 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:04:09.400548 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:04:09.400521 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d2b10b_88f6_464a_9828_9ad01eb27604.slice/crio-aa195b158753b50c79bda93b8c962c36c3e9a83744a765fe27504fd7691451f9 WatchSource:0}: Error finding container aa195b158753b50c79bda93b8c962c36c3e9a83744a765fe27504fd7691451f9: Status 404 returned error can't find the container with id aa195b158753b50c79bda93b8c962c36c3e9a83744a765fe27504fd7691451f9 Aug 05 18:04:09.403010 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.402950 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:09.403097 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.403062 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:09.403133 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.403103 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:09.733783 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.733747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerStarted","Data":"6f782f0ac32f7030e8935b6017758a058af082385aabaa084f82ec2ef4a26f0e"} Aug 05 18:04:09.734002 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.733807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerStarted","Data":"aa195b158753b50c79bda93b8c962c36c3e9a83744a765fe27504fd7691451f9"} Aug 05 18:04:09.754092 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:09.754042 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podStartSLOduration=1.754024362 podStartE2EDuration="1.754024362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:04:09.752424332 +0000 UTC m=+2683.183570600" watchObservedRunningTime="2026-08-05 18:04:09.754024362 +0000 UTC m=+2683.185170617" Aug 05 18:04:10.266515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:10.266480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:10.270891 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:10.270868 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:10.738921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:10.738892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:10.739918 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:10.739900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:11.743646 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:11.743614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:11.842846 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:11.842809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:11.943064 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:11.943035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:12.857039 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:12.857013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:13.184825 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:13.184736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:13.354203 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:13.354163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:13.439746 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:13.439667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:25.740100 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.740059 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:25.740642 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.740132 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:25.817023 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.816987 2564 generic.go:356] "Generic (PLEG): container finished" podID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerID="6f782f0ac32f7030e8935b6017758a058af082385aabaa084f82ec2ef4a26f0e" exitCode=0 Aug 05 18:04:25.817195 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.817060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerDied","Data":"6f782f0ac32f7030e8935b6017758a058af082385aabaa084f82ec2ef4a26f0e"} Aug 05 18:04:25.817583 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.817565 2564 scope.go:117] "RemoveContainer" containerID="6f782f0ac32f7030e8935b6017758a058af082385aabaa084f82ec2ef4a26f0e" Aug 05 18:04:25.818791 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.818756 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:25.818872 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.818852 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:25.818928 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:25.818911 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:26.823411 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:26.823379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerStarted","Data":"41b946ea67494cd8f2a8acee016c8171c386a1155017f673b78b05da81a3a50f"} Aug 05 18:04:27.266447 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:27.266417 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:27.266676 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:27.266656 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:27.266721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:27.266699 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:28.266168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:28.266135 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:28.266572 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:28.266199 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:29.265695 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:29.265661 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:29.265870 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:29.265723 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:30.266046 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:30.266012 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:30.266437 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:30.266068 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:31.266348 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:31.266309 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 18:04:31.266740 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:31.266367 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 18:04:32.269966 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:32.269920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:32.270416 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:32.270392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:32.271162 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:32.271144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:04:36.793653 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:36.793617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:37.255711 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.255622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:37.501706 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.501678 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:04:37.502048 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.502035 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:37.502102 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.502049 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:37.502102 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.502099 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03f0657-e32b-4b6b-802b-426367859f87" containerName="istio-proxy" Aug 05 18:04:37.503942 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.503924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.506813 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.506751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-adc09f-openshift-default-dockercfg-8dh57\"" Aug 05 18:04:37.517431 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.517409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:04:37.563062 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.563024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:37.613925 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.613886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpjf\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614392 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614392 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614392 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.614392 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.614293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715711 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715711 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpjf\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.715976 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.715944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.716341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.716229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.716391 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.716345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.716460 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.716388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.716460 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.716436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.716623 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.716606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.718343 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.718328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.718473 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.718457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.723727 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.723707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpjf\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.723828 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.723758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token\") pod \"e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.814770 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.814743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:37.945261 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.945225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:04:37.947044 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:04:37.947006 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c673cd_0180_40e2_80c4_e1bb0fad0e01.slice/crio-6b7ffff664ec532bd32a86f909bb65b2f76fce05c0478d3541457dfab5fdc808 WatchSource:0}: Error finding container 6b7ffff664ec532bd32a86f909bb65b2f76fce05c0478d3541457dfab5fdc808: Status 404 returned error can't find the container with id 6b7ffff664ec532bd32a86f909bb65b2f76fce05c0478d3541457dfab5fdc808 Aug 05 18:04:37.949085 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.949050 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:37.949194 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.949125 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:37.949194 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:37.949175 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:38.003032 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:38.003006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:38.880250 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:38.880217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerStarted","Data":"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e"} Aug 05 18:04:38.880713 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:38.880271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerStarted","Data":"6b7ffff664ec532bd32a86f909bb65b2f76fce05c0478d3541457dfab5fdc808"} Aug 05 18:04:38.901011 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:38.900928 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podStartSLOduration=1.9009109849999999 podStartE2EDuration="1.900910985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:04:38.899921696 +0000 UTC m=+2712.331067943" watchObservedRunningTime="2026-08-05 18:04:38.900910985 +0000 UTC m=+2712.332057237" Aug 05 18:04:39.815546 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:39.815509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:39.820558 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:39.820531 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:39.884867 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:39.884838 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:39.885719 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:39.885703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:40.847217 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:40.847184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:41.082183 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:41.082152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:41.981150 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:41.981118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:42.136849 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:42.136813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:42.574063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:42.574031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:42.707446 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:42.707410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:04:55.092884 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.092853 2564 patch_prober.go:28] interesting pod/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:52304->10.133.0.43:15021: read: connection reset by peer" start-of-body= Aug 05 18:04:55.093310 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.092971 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:52304->10.133.0.43:15021: read: connection reset by peer" Aug 05 18:04:55.961657 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.961617 2564 generic.go:356] "Generic (PLEG): container finished" podID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerID="8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e" exitCode=0 Aug 05 18:04:55.961829 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.961686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerDied","Data":"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e"} Aug 05 18:04:55.962150 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.962137 2564 scope.go:117] "RemoveContainer" containerID="8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e" Aug 05 18:04:55.963223 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.963193 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:55.963298 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.963282 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:55.963341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:55.963321 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:56.967692 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:56.967652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerStarted","Data":"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162"} Aug 05 18:04:57.814927 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:57.814887 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:04:57.815152 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:57.815126 2564 patch_prober.go:28] interesting pod/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 05 18:04:57.815214 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:57.815193 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 05 18:04:58.815814 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:58.815776 2564 patch_prober.go:28] interesting pod/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 05 18:04:58.816235 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:58.815847 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 05 18:04:59.815199 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:59.815166 2564 patch_prober.go:28] interesting pod/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 05 18:04:59.815386 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:04:59.815231 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 05 18:05:00.815735 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:00.815698 2564 patch_prober.go:28] interesting pod/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 05 18:05:00.816165 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:00.815753 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 05 18:05:01.819240 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:01.819213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:05:01.819615 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:01.819582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:05:01.820309 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:01.820293 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:05:05.855888 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:05.855855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:24.550535 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:24.550505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:24.636304 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:24.636272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:24.708563 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:24.708515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:24.771948 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:24.771915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:25.081530 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:25.081500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:30.666420 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:30.666387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:30.767972 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:30.767931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:30.947801 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:30.947724 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:05:30.947944 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:30.947922 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" containerID="cri-o://41b946ea67494cd8f2a8acee016c8171c386a1155017f673b78b05da81a3a50f" gracePeriod=30 Aug 05 18:05:30.973506 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:30.973477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:31.187875 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:31.187841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:32.273440 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:32.273400 2564 patch_prober.go:28] interesting pod/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:05:32.273903 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:32.273482 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:05:36.157724 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.157688 2564 generic.go:356] "Generic (PLEG): container finished" podID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerID="41b946ea67494cd8f2a8acee016c8171c386a1155017f673b78b05da81a3a50f" exitCode=0 Aug 05 18:05:36.158168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.157762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerDied","Data":"41b946ea67494cd8f2a8acee016c8171c386a1155017f673b78b05da81a3a50f"} Aug 05 18:05:36.158168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.157848 2564 scope.go:117] "RemoveContainer" containerID="6f782f0ac32f7030e8935b6017758a058af082385aabaa084f82ec2ef4a26f0e" Aug 05 18:05:36.206386 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.206363 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:05:36.268776 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.268776 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268719 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.268776 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.268776 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2jqx\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269152 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268803 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269152 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.268866 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269152 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269061 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:36.269152 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269106 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269357 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269157 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:36.269357 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269357 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data\") pod \"02d2b10b-88f6-464a-9828-9ad01eb27604\" (UID: \"02d2b10b-88f6-464a-9828-9ad01eb27604\") " Aug 05 18:05:36.269357 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269290 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:36.269357 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269314 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:05:36.269535 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269523 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.269583 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269537 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.269583 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269545 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.269583 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269554 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d2b10b-88f6-464a-9828-9ad01eb27604-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.269687 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.269543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data" (OuterVolumeSpecName: "istio-data") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:36.271048 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.271025 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:36.271329 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.271309 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token" (OuterVolumeSpecName: "istio-token") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:36.271510 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.271482 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:05:36.271619 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.271573 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx" (OuterVolumeSpecName: "kube-api-access-d2jqx") pod "02d2b10b-88f6-464a-9828-9ad01eb27604" (UID: "02d2b10b-88f6-464a-9828-9ad01eb27604"). InnerVolumeSpecName "kube-api-access-d2jqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:36.370843 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.370800 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.370843 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.370835 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.370843 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.370844 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.370843 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.370852 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d2b10b-88f6-464a-9828-9ad01eb27604-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:36.371161 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:36.370860 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jqx\" (UniqueName: \"kubernetes.io/projected/02d2b10b-88f6-464a-9828-9ad01eb27604-kube-api-access-d2jqx\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:37.163760 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:37.163732 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" Aug 05 18:05:37.164184 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:37.163726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn" event={"ID":"02d2b10b-88f6-464a-9828-9ad01eb27604","Type":"ContainerDied","Data":"aa195b158753b50c79bda93b8c962c36c3e9a83744a765fe27504fd7691451f9"} Aug 05 18:05:37.164184 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:37.163863 2564 scope.go:117] "RemoveContainer" containerID="41b946ea67494cd8f2a8acee016c8171c386a1155017f673b78b05da81a3a50f" Aug 05 18:05:37.182480 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:37.182448 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:05:37.186464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:37.186433 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d5bdd1-openshift-default-744c7d89fb-n5npn"] Aug 05 18:05:39.038756 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:39.038720 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" path="/var/lib/kubelet/pods/02d2b10b-88f6-464a-9828-9ad01eb27604/volumes" Aug 05 18:05:46.781294 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:46.781247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:46.862685 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:46.862656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:46.934875 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:46.934823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:47.011586 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:47.011537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:47.356368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:47.356336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:53.295511 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:53.295468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:53.601127 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:53.601074 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:05:53.601374 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:53.601329 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" containerID="cri-o://ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162" gracePeriod=30 Aug 05 18:05:54.546911 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.546881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:54.590944 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.590905 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:05:54.591294 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591280 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.591354 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591298 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.591354 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591318 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.591354 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591323 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.591451 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591382 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.591451 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.591393 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d2b10b-88f6-464a-9828-9ad01eb27604" containerName="istio-proxy" Aug 05 18:05:54.594275 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.594254 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.597592 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.597566 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-3f8e0d-openshift-default-dockercfg-jhbpj\"" Aug 05 18:05:54.610894 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.610862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:05:54.635627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.635591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:54.736153 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.736117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:54.752461 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6v6t\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752804 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752804 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752804 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.752923 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.752829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854204 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854204 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854204 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v6t\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854464 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854760 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854760 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.854760 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.854754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.855024 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.855003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.855309 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.855289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.856693 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.856673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.857046 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.857029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.862161 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.862132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.862161 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.862149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v6t\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t\") pod \"e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:54.904419 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:54.904381 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:55.055126 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.055100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:05:55.057016 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:05:55.056992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9503baed_d359_4676_8faf_b191e6cd405a.slice/crio-dbd002bd39a90b0666b0d4953d95121a089f4cfae352552df4b0a95c179d32a4 WatchSource:0}: Error finding container dbd002bd39a90b0666b0d4953d95121a089f4cfae352552df4b0a95c179d32a4: Status 404 returned error can't find the container with id dbd002bd39a90b0666b0d4953d95121a089f4cfae352552df4b0a95c179d32a4 Aug 05 18:05:55.059038 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.059004 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:55.059122 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.059073 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:55.059122 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.059110 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:55.252345 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.252314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerStarted","Data":"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610"} Aug 05 18:05:55.252655 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.252365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerStarted","Data":"dbd002bd39a90b0666b0d4953d95121a089f4cfae352552df4b0a95c179d32a4"} Aug 05 18:05:55.273164 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.273112 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podStartSLOduration=1.27309337 podStartE2EDuration="1.27309337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:05:55.271504686 +0000 UTC m=+2788.702650949" watchObservedRunningTime="2026-08-05 18:05:55.27309337 +0000 UTC m=+2788.704239619" Aug 05 18:05:55.905300 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:55.905262 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:56.909775 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:56.909744 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:57.184688 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:57.184607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:57.261618 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:57.261583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:57.262623 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:57.262604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:05:57.305715 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:57.305684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:58.483018 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.482980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:58.618974 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.618932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:58.855727 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.855699 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:05:58.940104 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.940074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:58.992417 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992381 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992417 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992550 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdpjf\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992639 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.992677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992675 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket\") pod \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\" (UID: \"93c673cd-0180-40e2-80c4-e1bb0fad0e01\") " Aug 05 18:05:58.993168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.992752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data" (OuterVolumeSpecName: "istio-data") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:58.993168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.993094 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:58.993355 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.993313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:58.993474 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.993389 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:58.993636 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.993601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:58.993800 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.993777 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:05:58.995237 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.995149 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:05:58.995366 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.995260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf" (OuterVolumeSpecName: "kube-api-access-rdpjf") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "kube-api-access-rdpjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:58.995491 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.995476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token" (OuterVolumeSpecName: "istio-token") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:58.995559 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:58.995544 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93c673cd-0180-40e2-80c4-e1bb0fad0e01" (UID: "93c673cd-0180-40e2-80c4-e1bb0fad0e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:59.046147 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.046114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:05:59.093744 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093711 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpjf\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-kube-api-access-rdpjf\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.093744 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093741 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.093744 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093750 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.094004 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093758 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.094004 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093768 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.094004 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093776 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.094004 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093785 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.094004 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.093792 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c673cd-0180-40e2-80c4-e1bb0fad0e01-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:05:59.271753 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.271720 2564 generic.go:356] "Generic (PLEG): container finished" podID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerID="ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162" exitCode=0 Aug 05 18:05:59.271921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.271771 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" Aug 05 18:05:59.271921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.271809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerDied","Data":"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162"} Aug 05 18:05:59.271921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.271871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g" event={"ID":"93c673cd-0180-40e2-80c4-e1bb0fad0e01","Type":"ContainerDied","Data":"6b7ffff664ec532bd32a86f909bb65b2f76fce05c0478d3541457dfab5fdc808"} Aug 05 18:05:59.271921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.271903 2564 scope.go:117] "RemoveContainer" containerID="ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162" Aug 05 18:05:59.280743 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.280726 2564 scope.go:117] "RemoveContainer" containerID="8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e" Aug 05 18:05:59.288766 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.288729 2564 scope.go:117] "RemoveContainer" containerID="ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162" Aug 05 18:05:59.289151 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:05:59.289122 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162\": container with ID starting with ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162 not found: ID does not exist" containerID="ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162" Aug 05 18:05:59.289262 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.289163 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162"} err="failed to get container status \"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162\": rpc error: code = NotFound desc = could not find container \"ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162\": container with ID starting with ab4922697ead45bc43fbd02bbb943ed39d8648e914da649cc8a789c2ca4d3162 not found: ID does not exist" Aug 05 18:05:59.289262 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.289188 2564 scope.go:117] "RemoveContainer" containerID="8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e" Aug 05 18:05:59.289475 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:05:59.289458 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e\": container with ID starting with 8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e not found: ID does not exist" containerID="8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e" Aug 05 18:05:59.289526 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.289483 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e"} err="failed to get container status \"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e\": rpc error: code = NotFound desc = could not find container \"8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e\": container with ID starting with 8463421911cf6cb79846474c5ee3b914ae355892097b245484dd64e42461997e not found: ID does not exist" Aug 05 18:05:59.290644 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.290627 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:05:59.295537 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:05:59.295512 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-adc09f-openshift-default-696df7cb7b-zns6g"] Aug 05 18:06:01.038165 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:01.038132 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" path="/var/lib/kubelet/pods/93c673cd-0180-40e2-80c4-e1bb0fad0e01/volumes" Aug 05 18:06:02.085696 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:02.085668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:11.338009 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.337947 2564 generic.go:356] "Generic (PLEG): container finished" podID="9503baed-d359-4676-8faf-b191e6cd405a" containerID="2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610" exitCode=0 Aug 05 18:06:11.338009 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.337988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerDied","Data":"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610"} Aug 05 18:06:11.338453 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.338375 2564 scope.go:117] "RemoveContainer" containerID="2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610" Aug 05 18:06:11.339362 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.339336 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:11.339421 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.339405 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:11.339461 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:11.339435 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:12.261536 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:12.261498 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:06:12.344703 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:12.344670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerStarted","Data":"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1"} Aug 05 18:06:12.905328 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:12.905283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:06:12.906435 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:12.906405 2564 patch_prober.go:28] interesting pod/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:06:12.906575 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:12.906456 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 05 18:06:13.905133 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:13.905089 2564 patch_prober.go:28] interesting pod/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:06:13.905564 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:13.905162 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 05 18:06:14.905741 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:14.905696 2564 patch_prober.go:28] interesting pod/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:06:14.906169 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:14.905764 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 05 18:06:15.905886 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:15.905848 2564 patch_prober.go:28] interesting pod/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:06:15.906311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:15.905945 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 05 18:06:16.905261 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:16.905219 2564 patch_prober.go:28] interesting pod/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:06:16.905449 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:16.905299 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Aug 05 18:06:17.909159 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:17.909128 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:06:17.909569 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:17.909375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:06:17.910171 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:17.910148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:06:22.172087 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:22.172046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:22.517232 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:22.517146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:22.816928 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:22.816888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:23.543337 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:23.543307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:25.260196 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:25.260166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:25.362715 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:25.362683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:26.394295 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:26.394240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:26.689568 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:26.689478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:26.903837 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:26.903807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:26.997835 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:26.997750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:40.043793 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:40.043752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:50.308791 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:50.308754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:06:55.397257 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:06:55.397222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:07:19.162909 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.162819 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:07:19.163370 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.163225 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:07:19.163370 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.163239 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:07:19.163370 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.163300 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:07:19.163370 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.163306 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:07:19.166226 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.166203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.168543 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.168524 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-30613e\"/\"kube-root-ca.crt\"" Aug 05 18:07:19.169743 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.169726 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-30613e\"/\"openshift-service-ca.crt\"" Aug 05 18:07:19.169836 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.169731 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-30613e\"/\"default-dockercfg-5zqkf\"" Aug 05 18:07:19.169836 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.169728 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-30613e\"/\"auth-test-model-30613e-kserve-self-signed-certs\"" Aug 05 18:07:19.176380 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.176361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:07:19.187325 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.187294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:07:19.234899 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.234864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.235120 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.234913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.235120 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.235009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.235120 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.235052 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.235120 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.235101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s5t\" (UniqueName: \"kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.235294 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.235127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336514 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336716 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336716 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336716 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336716 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s5t\" (UniqueName: \"kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.336716 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.337003 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.336929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.337080 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.337056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.337133 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.337065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.339029 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.339009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.339273 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.339256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.344119 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.344099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s5t\" (UniqueName: \"kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t\") pod \"auth-test-model-30613e-kserve-59778c875f-vhhhv\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.486896 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.486812 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:19.622066 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.622037 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:07:19.624845 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:07:19.624816 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeae06c80_9e80_4719_95f8_cf40c9c26d46.slice/crio-0c3ec291f4a7e456e28d2095a3ff8fce9952a116ec79b2e65f4db87d03da7625 WatchSource:0}: Error finding container 0c3ec291f4a7e456e28d2095a3ff8fce9952a116ec79b2e65f4db87d03da7625: Status 404 returned error can't find the container with id 0c3ec291f4a7e456e28d2095a3ff8fce9952a116ec79b2e65f4db87d03da7625 Aug 05 18:07:19.657897 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:19.657869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" event={"ID":"eae06c80-9e80-4719-95f8-cf40c9c26d46","Type":"ContainerStarted","Data":"0c3ec291f4a7e456e28d2095a3ff8fce9952a116ec79b2e65f4db87d03da7625"} Aug 05 18:07:20.663885 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:20.663851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" event={"ID":"eae06c80-9e80-4719-95f8-cf40c9c26d46","Type":"ContainerStarted","Data":"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e"} Aug 05 18:07:20.682084 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:20.682040 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" podStartSLOduration=1.68202514 podStartE2EDuration="1.68202514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:07:20.680170746 +0000 UTC m=+2874.111317000" watchObservedRunningTime="2026-08-05 18:07:20.68202514 +0000 UTC m=+2874.113171388" Aug 05 18:07:29.487189 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:29.487151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:29.487736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:29.487212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:29.499624 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:29.499596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:07:29.716521 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:07:29.716482 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:10:01.674046 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:01.674006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:10:15.723176 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:15.723144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:10:16.557552 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:16.557523 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:10:16.607231 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:16.607195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:10:20.487318 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:20.487237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:10:27.285172 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:27.285143 2564 scope.go:117] "RemoveContainer" containerID="5469c2ed57c6a203a9dff5d450bd006ba65f6ee740506b1426addef246d02525" Aug 05 18:10:31.578799 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.578734 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" podUID="eae06c80-9e80-4719-95f8-cf40c9c26d46" containerName="main" containerID="cri-o://befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e" gracePeriod=45 Aug 05 18:10:31.834175 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.834110 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:10:31.991859 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.991829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.991859 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.991866 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.992089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.991918 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.992089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.991946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.992089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7s5t\" (UniqueName: \"kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.992089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache\") pod \"eae06c80-9e80-4719-95f8-cf40c9c26d46\" (UID: \"eae06c80-9e80-4719-95f8-cf40c9c26d46\") " Aug 05 18:10:31.992286 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:10:31.992286 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home" (OuterVolumeSpecName: "home") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:10:31.992394 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992368 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-home\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:31.992445 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992396 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache" (OuterVolumeSpecName: "model-cache") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:10:31.992445 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.992421 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-tmp-dir\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:31.994311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.994289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:10:31.994311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.994298 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm" (OuterVolumeSpecName: "dshm") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:10:31.994478 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:31.994315 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t" (OuterVolumeSpecName: "kube-api-access-h7s5t") pod "eae06c80-9e80-4719-95f8-cf40c9c26d46" (UID: "eae06c80-9e80-4719-95f8-cf40c9c26d46"). InnerVolumeSpecName "kube-api-access-h7s5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:10:32.093301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.093223 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-dshm\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:32.093301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.093247 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae06c80-9e80-4719-95f8-cf40c9c26d46-tls-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:32.093301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.093257 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7s5t\" (UniqueName: \"kubernetes.io/projected/eae06c80-9e80-4719-95f8-cf40c9c26d46-kube-api-access-h7s5t\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:32.093301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.093265 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae06c80-9e80-4719-95f8-cf40c9c26d46-model-cache\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:10:32.534130 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.534038 2564 generic.go:356] "Generic (PLEG): container finished" podID="eae06c80-9e80-4719-95f8-cf40c9c26d46" containerID="befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e" exitCode=0 Aug 05 18:10:32.534130 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.534083 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" Aug 05 18:10:32.534368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.534127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" event={"ID":"eae06c80-9e80-4719-95f8-cf40c9c26d46","Type":"ContainerDied","Data":"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e"} Aug 05 18:10:32.534368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.534178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv" event={"ID":"eae06c80-9e80-4719-95f8-cf40c9c26d46","Type":"ContainerDied","Data":"0c3ec291f4a7e456e28d2095a3ff8fce9952a116ec79b2e65f4db87d03da7625"} Aug 05 18:10:32.534368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.534199 2564 scope.go:117] "RemoveContainer" containerID="befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e" Aug 05 18:10:32.542826 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.542810 2564 scope.go:117] "RemoveContainer" containerID="befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e" Aug 05 18:10:32.543079 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:10:32.543063 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e\": container with ID starting with befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e not found: ID does not exist" containerID="befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e" Aug 05 18:10:32.543133 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.543086 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e"} err="failed to get container status \"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e\": rpc error: code = NotFound desc = could not find container \"befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e\": container with ID starting with befc6002ee15c578308c4b409203bb89c7f7e20ceb010a973c5274e64776447e not found: ID does not exist" Aug 05 18:10:32.554804 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.554780 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:10:32.558415 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:32.558394 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-30613e/auth-test-model-30613e-kserve-59778c875f-vhhhv"] Aug 05 18:10:33.038504 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:33.038478 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae06c80-9e80-4719-95f8-cf40c9c26d46" path="/var/lib/kubelet/pods/eae06c80-9e80-4719-95f8-cf40c9c26d46/volumes" Aug 05 18:10:46.002349 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:10:46.002311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:11:16.760327 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:11:16.760297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:11:18.425896 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:11:18.425863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:11:52.127374 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:11:52.127275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:11:52.996698 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:11:52.996656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:12:08.073056 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073022 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:12:08.073438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073350 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae06c80-9e80-4719-95f8-cf40c9c26d46" containerName="main" Aug 05 18:12:08.073438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073359 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae06c80-9e80-4719-95f8-cf40c9c26d46" containerName="main" Aug 05 18:12:08.073438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:12:08.073438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073381 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c673cd-0180-40e2-80c4-e1bb0fad0e01" containerName="istio-proxy" Aug 05 18:12:08.073438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.073435 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae06c80-9e80-4719-95f8-cf40c9c26d46" containerName="main" Aug 05 18:12:08.076244 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.076228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.078700 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.078680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f8e0d\"/\"rate-test-model-3f8e0d-kserve-self-signed-certs\"" Aug 05 18:12:08.079780 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.079764 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f8e0d\"/\"openshift-service-ca.crt\"" Aug 05 18:12:08.080112 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.080096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f8e0d\"/\"default-dockercfg-xpvxf\"" Aug 05 18:12:08.080173 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.080160 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f8e0d\"/\"kube-root-ca.crt\"" Aug 05 18:12:08.086781 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.086761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:12:08.156285 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.156410 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2c7\" (UniqueName: \"kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.156458 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.156458 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.156532 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.156532 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.156499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.158189 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.158167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:12:08.257746 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.257712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.257908 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.257763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.257908 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.257793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258067 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258127 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258182 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258242 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258242 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2c7\" (UniqueName: \"kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.258341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.258260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.260723 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.260691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.261302 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.261252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.266945 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.266921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2c7\" (UniqueName: \"kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7\") pod \"rate-test-model-3f8e0d-kserve-665779cb8c-6888w\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.401062 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.400985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:08.532887 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.532860 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:12:08.535473 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:12:08.535441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf44a786a_5ea7_4306_8edc_c11bd27f5196.slice/crio-b67504f9a86aa35e61cad16c8df1e54268fb29c505b493bc1574f1d62b62ddb1 WatchSource:0}: Error finding container b67504f9a86aa35e61cad16c8df1e54268fb29c505b493bc1574f1d62b62ddb1: Status 404 returned error can't find the container with id b67504f9a86aa35e61cad16c8df1e54268fb29c505b493bc1574f1d62b62ddb1 Aug 05 18:12:08.958526 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.958489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" event={"ID":"f44a786a-5ea7-4306-8edc-c11bd27f5196","Type":"ContainerStarted","Data":"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43"} Aug 05 18:12:08.958710 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.958538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" event={"ID":"f44a786a-5ea7-4306-8edc-c11bd27f5196","Type":"ContainerStarted","Data":"b67504f9a86aa35e61cad16c8df1e54268fb29c505b493bc1574f1d62b62ddb1"} Aug 05 18:12:08.977438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:08.977392 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" podStartSLOduration=0.977378971 podStartE2EDuration="977.378971ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:12:08.97508968 +0000 UTC m=+3162.406235931" watchObservedRunningTime="2026-08-05 18:12:08.977378971 +0000 UTC m=+3162.408525220" Aug 05 18:12:18.401799 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:18.401763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:18.402225 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:18.401823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:18.413870 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:18.413848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:19.016685 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:19.016657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:12:27.264338 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:27.264301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:12:31.507058 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:12:31.507021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:35.022828 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:35.022791 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:13:35.045983 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:35.045938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:36.005627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:36.005594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:50.042834 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.042790 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" podUID="f44a786a-5ea7-4306-8edc-c11bd27f5196" containerName="main" containerID="cri-o://93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43" gracePeriod=45 Aug 05 18:13:50.285183 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.285155 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:13:50.352913 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.352828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg2c7\" (UniqueName: \"kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.352913 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.352882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.353149 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.352920 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.353149 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.352970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.353149 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353020 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.353149 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353078 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache\") pod \"f44a786a-5ea7-4306-8edc-c11bd27f5196\" (UID: \"f44a786a-5ea7-4306-8edc-c11bd27f5196\") " Aug 05 18:13:50.353366 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353221 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:50.353366 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353330 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home" (OuterVolumeSpecName: "home") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:50.353468 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353371 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache" (OuterVolumeSpecName: "model-cache") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:50.353468 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353441 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-tmp-dir\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:50.353468 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.353461 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-home\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:50.355214 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.355195 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm" (OuterVolumeSpecName: "dshm") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:50.355281 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.355261 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7" (OuterVolumeSpecName: "kube-api-access-sg2c7") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "kube-api-access-sg2c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:50.355367 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.355348 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f44a786a-5ea7-4306-8edc-c11bd27f5196" (UID: "f44a786a-5ea7-4306-8edc-c11bd27f5196"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:13:50.426063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.426027 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44a786a-5ea7-4306-8edc-c11bd27f5196" containerID="93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43" exitCode=0 Aug 05 18:13:50.426063 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.426067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" event={"ID":"f44a786a-5ea7-4306-8edc-c11bd27f5196","Type":"ContainerDied","Data":"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43"} Aug 05 18:13:50.426311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.426073 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" Aug 05 18:13:50.426311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.426102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w" event={"ID":"f44a786a-5ea7-4306-8edc-c11bd27f5196","Type":"ContainerDied","Data":"b67504f9a86aa35e61cad16c8df1e54268fb29c505b493bc1574f1d62b62ddb1"} Aug 05 18:13:50.426311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.426121 2564 scope.go:117] "RemoveContainer" containerID="93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43" Aug 05 18:13:50.434775 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.434758 2564 scope.go:117] "RemoveContainer" containerID="93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43" Aug 05 18:13:50.435038 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:13:50.435018 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43\": container with ID starting with 93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43 not found: ID does not exist" containerID="93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43" Aug 05 18:13:50.435101 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.435046 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43"} err="failed to get container status \"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43\": rpc error: code = NotFound desc = could not find container \"93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43\": container with ID starting with 93564e4fbd852136ca9f23830f63f0fc817425496879c6e38f46250b8532bd43 not found: ID does not exist" Aug 05 18:13:50.449017 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.448989 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:13:50.453668 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.453648 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f8e0d/rate-test-model-3f8e0d-kserve-665779cb8c-6888w"] Aug 05 18:13:50.454066 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.454049 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-model-cache\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:50.454118 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.454069 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2c7\" (UniqueName: \"kubernetes.io/projected/f44a786a-5ea7-4306-8edc-c11bd27f5196-kube-api-access-sg2c7\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:50.454118 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.454079 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f44a786a-5ea7-4306-8edc-c11bd27f5196-dshm\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:50.454118 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:50.454088 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f44a786a-5ea7-4306-8edc-c11bd27f5196-tls-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:13:51.039060 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:51.039030 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44a786a-5ea7-4306-8edc-c11bd27f5196" path="/var/lib/kubelet/pods/f44a786a-5ea7-4306-8edc-c11bd27f5196/volumes" Aug 05 18:13:55.691359 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.691323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:55.976343 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.976244 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:13:55.976792 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.976760 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44a786a-5ea7-4306-8edc-c11bd27f5196" containerName="main" Aug 05 18:13:55.976792 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.976783 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44a786a-5ea7-4306-8edc-c11bd27f5196" containerName="main" Aug 05 18:13:55.977047 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.976874 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44a786a-5ea7-4306-8edc-c11bd27f5196" containerName="main" Aug 05 18:13:55.981530 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.981504 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:55.983981 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.983944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4babb1-openshift-default-dockercfg-h56qx\"" Aug 05 18:13:55.992723 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:55.992638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:13:56.002807 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.002766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:56.112502 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112502 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112764 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf2c\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112764 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112764 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112764 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.112764 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.113043 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.113043 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.112876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.204872 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.204839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:56.214143 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214215 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214215 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214285 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214446 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214519 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214577 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214577 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf2c\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214682 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214733 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214787 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.214993 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.214974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.215259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.215240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.216830 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.216805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.216921 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.216854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.222265 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.222244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.222348 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.222294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf2c\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c\") pod \"e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.292934 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.292911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:56.432470 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.432433 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:13:56.434386 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:13:56.434357 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e576beb_e550_4696_bc4b_c948e6ea6f6a.slice/crio-294720bc75b0ea7f1d2a7bf14118def80550ef22d7ca767f428f569f374bfc0b WatchSource:0}: Error finding container 294720bc75b0ea7f1d2a7bf14118def80550ef22d7ca767f428f569f374bfc0b: Status 404 returned error can't find the container with id 294720bc75b0ea7f1d2a7bf14118def80550ef22d7ca767f428f569f374bfc0b Aug 05 18:13:56.436669 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.436638 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:13:56.436739 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.436705 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:13:56.436739 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.436732 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:13:56.459553 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:56.459522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerStarted","Data":"294720bc75b0ea7f1d2a7bf14118def80550ef22d7ca767f428f569f374bfc0b"} Aug 05 18:13:57.465372 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:57.465336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerStarted","Data":"7e350f0175fb5f5c220d2672aa239b903a14e7485d2d6600b3591f688ff7bda1"} Aug 05 18:13:57.488150 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:57.488098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podStartSLOduration=2.488081496 podStartE2EDuration="2.488081496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:13:57.486302498 +0000 UTC m=+3270.917448747" watchObservedRunningTime="2026-08-05 18:13:57.488081496 +0000 UTC m=+3270.919227745" Aug 05 18:13:58.293651 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:58.293617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:58.298515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:58.298491 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:58.470415 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:58.470386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:58.471341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:58.471324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:13:59.523039 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:59.523011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:13:59.668841 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:13:59.668808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:00.715773 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:00.715729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:00.837155 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:00.837129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:01.176833 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:01.176799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:01.274810 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:01.274776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:12.548345 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.548308 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerID="7e350f0175fb5f5c220d2672aa239b903a14e7485d2d6600b3591f688ff7bda1" exitCode=0 Aug 05 18:14:12.548779 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.548357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerDied","Data":"7e350f0175fb5f5c220d2672aa239b903a14e7485d2d6600b3591f688ff7bda1"} Aug 05 18:14:12.548835 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.548797 2564 scope.go:117] "RemoveContainer" containerID="7e350f0175fb5f5c220d2672aa239b903a14e7485d2d6600b3591f688ff7bda1" Aug 05 18:14:12.550001 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.549969 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:12.550101 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.550039 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:12.550101 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:12.550067 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:13.470821 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:13.470787 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:14:13.554159 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:13.554122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerStarted","Data":"c88c9ae911013e2bb9d695f0f2851ce9b06dea1b04036e22c45249f73a0b02de"} Aug 05 18:14:14.293478 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:14.293395 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:14:14.294687 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:14.294663 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 05 18:14:14.294893 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:14.294870 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 05 18:14:15.294980 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:15.294248 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 05 18:14:15.294980 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:15.294306 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 05 18:14:16.294356 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:16.294321 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 05 18:14:16.294627 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:16.294597 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 05 18:14:17.293420 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:17.293387 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Aug 05 18:14:17.293997 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:17.293945 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Aug 05 18:14:18.297695 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:18.297664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:14:18.298144 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:18.297927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:14:18.298647 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:18.298629 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:14:24.450225 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:24.450192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:25.157391 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.157360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:25.430099 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.430025 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:14:25.432392 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.432372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.435053 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.435020 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a83758-openshift-default-dockercfg-4xs25\"" Aug 05 18:14:25.444516 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.444487 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:14:25.463115 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.463095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:25.478209 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478310 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478310 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478310 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478444 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478444 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslgn\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478444 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478444 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.478444 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.478422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579321 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579487 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslgn\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579487 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579487 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579487 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579487 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579707 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579707 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.579707 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.579572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.580180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.580139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.580180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.580147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.580533 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.580209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.580533 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.580220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.580533 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.580401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.582315 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.582294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.582401 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.582314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.591038 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.591016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslgn\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.591131 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.591089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token\") pod \"e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.626344 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.626317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:25.743405 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.743340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:25.877687 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.877662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:14:25.879241 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:14:25.879213 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3794d9_5d3e_4f6f_9535_7ceca6dd6b37.slice/crio-5a29f221f14ec073b89289b9d1081791ceada3cdc8d953d8367b5e9206670994 WatchSource:0}: Error finding container 5a29f221f14ec073b89289b9d1081791ceada3cdc8d953d8367b5e9206670994: Status 404 returned error can't find the container with id 5a29f221f14ec073b89289b9d1081791ceada3cdc8d953d8367b5e9206670994 Aug 05 18:14:25.881515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.881486 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:25.881602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.881546 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:25.881602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:25.881573 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:26.616222 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:26.616189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerStarted","Data":"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12"} Aug 05 18:14:26.616600 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:26.616242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerStarted","Data":"5a29f221f14ec073b89289b9d1081791ceada3cdc8d953d8367b5e9206670994"} Aug 05 18:14:26.637390 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:26.637343 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podStartSLOduration=1.637328981 podStartE2EDuration="1.637328981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:14:26.635464809 +0000 UTC m=+3300.066611074" watchObservedRunningTime="2026-08-05 18:14:26.637328981 +0000 UTC m=+3300.068475231" Aug 05 18:14:26.743504 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:26.743480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:26.749511 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:26.749480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:27.621052 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:27.620977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:27.621823 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:27.621804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:28.564442 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:28.564405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:28.661112 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:28.661086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:29.699405 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:29.699376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:29.763053 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:29.763023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:30.199848 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:30.199809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:30.303497 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:30.303466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:14:42.622027 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.621999 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:42.622359 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.622052 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:42.691451 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.691399 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerID="e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12" exitCode=0 Aug 05 18:14:42.691550 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.691455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerDied","Data":"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12"} Aug 05 18:14:42.691781 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.691763 2564 scope.go:117] "RemoveContainer" containerID="e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12" Aug 05 18:14:42.692818 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.692786 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:42.692906 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.692875 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:42.692987 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:42.692924 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:43.701262 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:43.701214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerStarted","Data":"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5"} Aug 05 18:14:43.743742 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:43.743711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:43.743994 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:43.743951 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:43.744050 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:43.744019 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:44.743784 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:44.743752 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:44.744170 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:44.743809 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:45.743894 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:45.743862 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:45.744284 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:45.743911 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:46.744575 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:46.744543 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:46.744984 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:46.744594 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:47.744482 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:47.744402 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 18:14:47.744482 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:47.744460 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 18:14:48.748056 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:48.748029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:48.748458 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:48.748293 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:48.748914 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:48.748894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:14:54.193767 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:14:54.193706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:03.619663 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:03.619630 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5"] Aug 05 18:15:03.621400 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:03.621378 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765850-pkdc5"] Aug 05 18:15:05.038084 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:05.038050 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a7978c-ad50-4e29-9a1d-3aaf56212930" path="/var/lib/kubelet/pods/20a7978c-ad50-4e29-9a1d-3aaf56212930/volumes" Aug 05 18:15:07.582643 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:07.582612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:12.691211 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:12.691175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:27.336973 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:27.336928 2564 scope.go:117] "RemoveContainer" containerID="a0fc795253cfabd7d2160bb2005f358fb0d68bfa26c54b36307cd1429d9864ff" Aug 05 18:15:50.914263 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:50.914214 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:51.193759 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.193678 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:15:51.196153 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.196131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.200176 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.200154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-fb84d0-openshift-default-dockercfg-vfpfv\"" Aug 05 18:15:51.209562 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.209532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:15:51.230119 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.230092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:51.285569 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285685 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8968\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285685 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285782 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285782 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285782 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285909 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285909 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.285909 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.285878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.331112 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.331089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:51.387089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.387759 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.387759 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.387972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.388019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388342 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.388054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8968\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388517 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.388493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.388818 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.388796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.389019 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.388996 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.389878 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.389856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.391452 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.391185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.391566 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.391495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.397434 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.397410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8968\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.398877 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.398855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token\") pod \"e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.506180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.506103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:51.636553 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.636529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:15:51.638373 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:15:51.638344 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda950a493_0bda_4cfd_8979_2e6caf43a257.slice/crio-60e1c1c6b58a1f1c58e9a3990dc813294dc5f36ee900f882cde9f0c6d90d43a0 WatchSource:0}: Error finding container 60e1c1c6b58a1f1c58e9a3990dc813294dc5f36ee900f882cde9f0c6d90d43a0: Status 404 returned error can't find the container with id 60e1c1c6b58a1f1c58e9a3990dc813294dc5f36ee900f882cde9f0c6d90d43a0 Aug 05 18:15:51.640679 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.640636 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:51.640771 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.640749 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:51.640812 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:51.640802 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:52.006275 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:52.006242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerStarted","Data":"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d"} Aug 05 18:15:52.006641 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:52.006321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerStarted","Data":"60e1c1c6b58a1f1c58e9a3990dc813294dc5f36ee900f882cde9f0c6d90d43a0"} Aug 05 18:15:52.026922 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:52.026881 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podStartSLOduration=1.02686752 podStartE2EDuration="1.02686752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:15:52.025691154 +0000 UTC m=+3385.456837426" watchObservedRunningTime="2026-08-05 18:15:52.02686752 +0000 UTC m=+3385.458013770" Aug 05 18:15:52.507000 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:52.506944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:52.511659 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:52.511639 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:53.013211 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:53.013185 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:53.014132 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:53.014114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:15:53.921677 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:53.921649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:54.021296 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:54.021271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:55.051975 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:55.051935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:55.277023 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:55.276948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:55.646728 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:55.646698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:15:55.742546 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:15:55.742506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:09.055084 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:09.055054 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 05 18:16:09.055084 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:09.055097 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 05 18:16:10.088035 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.087978 2564 generic.go:356] "Generic (PLEG): container finished" podID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerID="8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d" exitCode=0 Aug 05 18:16:10.088339 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.088031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerDied","Data":"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d"} Aug 05 18:16:10.088409 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.088395 2564 scope.go:117] "RemoveContainer" containerID="8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d" Aug 05 18:16:10.089429 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.089401 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:16:10.089489 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.089475 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:16:10.089523 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:10.089514 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:16:11.098023 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:11.097984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerStarted","Data":"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326"} Aug 05 18:16:11.507180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:11.507078 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:11.507364 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:11.507312 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 05 18:16:11.507434 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:11.507367 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 05 18:16:12.507261 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:12.507228 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 05 18:16:12.507625 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:12.507299 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 05 18:16:13.507167 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:13.507133 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 05 18:16:13.507337 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:13.507183 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 05 18:16:14.506786 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:14.506753 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 05 18:16:14.506948 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:14.506805 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 05 18:16:15.507454 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:15.507420 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Aug 05 18:16:15.507852 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:15.507471 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Aug 05 18:16:16.509872 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:16.509846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:16.510333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:16.510212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:16.510877 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:16.510858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:19.429806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:19.429720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:25.185024 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:25.184948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:25.286252 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:25.286226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:25.356284 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:25.356256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:25.427214 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:25.427182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:25.917068 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:25.917036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:31.711210 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:31.711182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:31.790440 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:31.790411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:31.962912 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:31.962831 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:16:31.963121 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:31.963096 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" containerID="cri-o://6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326" gracePeriod=30 Aug 05 18:16:31.992141 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:31.992111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:32.220558 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:32.220473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:37.215881 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.215850 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:37.219327 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.219303 2564 generic.go:356] "Generic (PLEG): container finished" podID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerID="6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326" exitCode=0 Aug 05 18:16:37.219439 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.219345 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" Aug 05 18:16:37.219439 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.219379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerDied","Data":"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326"} Aug 05 18:16:37.219439 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.219423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s" event={"ID":"a950a493-0bda-4cfd-8979-2e6caf43a257","Type":"ContainerDied","Data":"60e1c1c6b58a1f1c58e9a3990dc813294dc5f36ee900f882cde9f0c6d90d43a0"} Aug 05 18:16:37.219547 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.219444 2564 scope.go:117] "RemoveContainer" containerID="6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326" Aug 05 18:16:37.227644 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.227628 2564 scope.go:117] "RemoveContainer" containerID="8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d" Aug 05 18:16:37.235101 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.235086 2564 scope.go:117] "RemoveContainer" containerID="6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326" Aug 05 18:16:37.235339 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:16:37.235311 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326\": container with ID starting with 6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326 not found: ID does not exist" containerID="6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326" Aug 05 18:16:37.235411 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.235347 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326"} err="failed to get container status \"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326\": rpc error: code = NotFound desc = could not find container \"6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326\": container with ID starting with 6188a29c99d03534c8f94b7005a4c00fe8c559335d751ce049befc573f7aa326 not found: ID does not exist" Aug 05 18:16:37.235411 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.235363 2564 scope.go:117] "RemoveContainer" containerID="8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d" Aug 05 18:16:37.235613 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:16:37.235596 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d\": container with ID starting with 8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d not found: ID does not exist" containerID="8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d" Aug 05 18:16:37.235706 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.235613 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d"} err="failed to get container status \"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d\": rpc error: code = NotFound desc = could not find container \"8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d\": container with ID starting with 8019fac3eff67a03075398c9927ebfc3b6c8d68a03fd9be28933e37200e9ff0d not found: ID does not exist" Aug 05 18:16:37.304864 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.304835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.304864 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.304869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.304893 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.304910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8968\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.304935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305079 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305139 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305467 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data\") pod \"a950a493-0bda-4cfd-8979-2e6caf43a257\" (UID: \"a950a493-0bda-4cfd-8979-2e6caf43a257\") " Aug 05 18:16:37.305467 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305198 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:16:37.305467 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:16:37.305467 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305429 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:16:37.305684 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305535 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.305684 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305554 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.305684 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305569 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.305684 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305531 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data" (OuterVolumeSpecName: "istio-data") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:16:37.305684 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.305549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:16:37.307277 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.307250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:16:37.307372 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.307304 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:16:37.307372 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.307348 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token" (OuterVolumeSpecName: "istio-token") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:16:37.307489 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.307474 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968" (OuterVolumeSpecName: "kube-api-access-g8968") pod "a950a493-0bda-4cfd-8979-2e6caf43a257" (UID: "a950a493-0bda-4cfd-8979-2e6caf43a257"). InnerVolumeSpecName "kube-api-access-g8968". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:16:37.406410 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406378 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.406410 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406404 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8968\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-kube-api-access-g8968\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.406410 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406416 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a950a493-0bda-4cfd-8979-2e6caf43a257-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.406623 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406424 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.406623 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406433 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.406623 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.406440 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a950a493-0bda-4cfd-8979-2e6caf43a257-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:16:37.540259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.540228 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:16:37.545882 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:37.545843 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fb84d0-openshift-default-7b6c95b44d-n9m6s"] Aug 05 18:16:39.038480 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:39.038448 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" path="/var/lib/kubelet/pods/a950a493-0bda-4cfd-8979-2e6caf43a257/volumes" Aug 05 18:16:51.216917 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:51.216881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:51.298429 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:51.298400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:51.385806 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:51.385767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:51.454717 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:51.454684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:51.794942 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:51.794911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:16:56.871238 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:16:56.871206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:17:03.349111 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:17:03.349072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:17:33.201909 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:17:33.201874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:17:33.801734 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:17:33.801699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:17:39.260009 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:17:39.259973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:02.188249 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.188175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:02.289253 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.289221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:02.432169 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.432137 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:18:02.432438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.432400 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" containerID="cri-o://c88c9ae911013e2bb9d695f0f2851ce9b06dea1b04036e22c45249f73a0b02de" gracePeriod=30 Aug 05 18:18:02.461184 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.461112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:02.711089 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:02.711059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:03.301040 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:03.301003 2564 patch_prober.go:28] interesting pod/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:18:03.301450 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:03.301072 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:18:07.639415 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.639383 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerID="c88c9ae911013e2bb9d695f0f2851ce9b06dea1b04036e22c45249f73a0b02de" exitCode=0 Aug 05 18:18:07.639860 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.639461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerDied","Data":"c88c9ae911013e2bb9d695f0f2851ce9b06dea1b04036e22c45249f73a0b02de"} Aug 05 18:18:07.639860 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.639545 2564 scope.go:117] "RemoveContainer" containerID="7e350f0175fb5f5c220d2672aa239b903a14e7485d2d6600b3591f688ff7bda1" Aug 05 18:18:07.684136 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.684115 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:18:07.779278 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779189 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779278 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779271 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779411 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779449 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779528 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgf2c\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779542 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:07.779725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779581 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:07.779725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779725 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data\") pod \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\" (UID: \"5e576beb-e550-4696-bc4b-c948e6ea6f6a\") " Aug 05 18:18:07.779991 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779810 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:07.779991 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.779857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:18:07.780079 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:07.780362 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780344 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.780407 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780369 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.780407 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780383 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.780407 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780395 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.780511 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.780408 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.781758 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.781731 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:07.781868 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.781731 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:07.782025 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.782005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:18:07.782098 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.782026 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c" (OuterVolumeSpecName: "kube-api-access-zgf2c") pod "5e576beb-e550-4696-bc4b-c948e6ea6f6a" (UID: "5e576beb-e550-4696-bc4b-c948e6ea6f6a"). InnerVolumeSpecName "kube-api-access-zgf2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:07.881795 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.881763 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf2c\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-kube-api-access-zgf2c\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.881795 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.881789 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.881795 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.881798 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:07.882051 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:07.881806 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e576beb-e550-4696-bc4b-c948e6ea6f6a-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:08.645602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:08.645564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" event={"ID":"5e576beb-e550-4696-bc4b-c948e6ea6f6a","Type":"ContainerDied","Data":"294720bc75b0ea7f1d2a7bf14118def80550ef22d7ca767f428f569f374bfc0b"} Aug 05 18:18:08.645602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:08.645593 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679" Aug 05 18:18:08.646167 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:08.645633 2564 scope.go:117] "RemoveContainer" containerID="c88c9ae911013e2bb9d695f0f2851ce9b06dea1b04036e22c45249f73a0b02de" Aug 05 18:18:08.667739 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:08.667708 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:18:08.671071 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:08.671048 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4babb1-openshift-default-58f744db54-hx679"] Aug 05 18:18:09.039208 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:09.039123 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" path="/var/lib/kubelet/pods/5e576beb-e550-4696-bc4b-c948e6ea6f6a/volumes" Aug 05 18:18:10.728301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:10.728257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:13.264513 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:13.264484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:13.381403 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:13.381365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:13.574945 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:13.574907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:13.831466 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:13.831384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:19.870197 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:19.870161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:19.993902 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:19.993850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:20.115427 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:20.115391 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:18:20.115636 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:20.115613 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" containerID="cri-o://3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5" gracePeriod=30 Aug 05 18:18:20.137712 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:20.137612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:20.359592 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:20.359553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:25.371521 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.371498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:18:25.455048 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.454924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455048 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.454989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455048 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455026 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455104 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455161 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455205 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455237 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455262 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:25.455333 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qslgn\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn\") pod \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\" (UID: \"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37\") " Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455498 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455502 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455595 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455657 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455675 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455689 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455702 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.455721 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.455711 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.457438 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.457415 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:25.457652 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.457638 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn" (OuterVolumeSpecName: "kube-api-access-qslgn") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "kube-api-access-qslgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:25.457762 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.457745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:18:25.457809 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.457776 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" (UID: "fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:25.556227 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.556195 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.556227 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.556221 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.556227 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.556231 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.556462 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.556239 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslgn\" (UniqueName: \"kubernetes.io/projected/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37-kube-api-access-qslgn\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:25.723632 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.723539 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerID="3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5" exitCode=0 Aug 05 18:18:25.723632 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.723604 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" Aug 05 18:18:25.723863 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.723619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerDied","Data":"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5"} Aug 05 18:18:25.723863 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.723713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j" event={"ID":"fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37","Type":"ContainerDied","Data":"5a29f221f14ec073b89289b9d1081791ceada3cdc8d953d8367b5e9206670994"} Aug 05 18:18:25.723863 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.723745 2564 scope.go:117] "RemoveContainer" containerID="3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5" Aug 05 18:18:25.732971 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.732940 2564 scope.go:117] "RemoveContainer" containerID="e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12" Aug 05 18:18:25.740842 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.740822 2564 scope.go:117] "RemoveContainer" containerID="3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5" Aug 05 18:18:25.741155 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:18:25.741137 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5\": container with ID starting with 3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5 not found: ID does not exist" containerID="3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5" Aug 05 18:18:25.741207 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.741164 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5"} err="failed to get container status \"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5\": rpc error: code = NotFound desc = could not find container \"3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5\": container with ID starting with 3e690a876d91a7005dcd9067f140e1d418be23ad2aea87de5d55269fefd530f5 not found: ID does not exist" Aug 05 18:18:25.741207 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.741182 2564 scope.go:117] "RemoveContainer" containerID="e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12" Aug 05 18:18:25.741429 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:18:25.741412 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12\": container with ID starting with e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12 not found: ID does not exist" containerID="e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12" Aug 05 18:18:25.741473 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.741444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12"} err="failed to get container status \"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12\": rpc error: code = NotFound desc = could not find container \"e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12\": container with ID starting with e1e1b67dacc4969692a531f38f79a5be41c1042071b125614c7ed2df3d546f12 not found: ID does not exist" Aug 05 18:18:25.747366 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.747338 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:18:25.750548 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:25.750522 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a83758-openshift-default-869cb8d5dd-jql7j"] Aug 05 18:18:27.039643 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:27.039612 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" path="/var/lib/kubelet/pods/fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37/volumes" Aug 05 18:18:30.949784 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:30.949751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:31.025515 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:31.025481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:31.124673 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:31.124638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:31.198013 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:31.197979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:31.515093 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:31.515058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:33.437449 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:33.437404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:33.535388 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:33.535349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:37.633738 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:37.633704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:37.736168 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:37.736126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:37.780073 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:37.780039 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:18:37.780366 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:37.780331 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" containerID="cri-o://e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1" gracePeriod=30 Aug 05 18:18:37.803136 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:37.803102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:38.066871 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:38.066820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:43.028509 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.028485 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:18:43.033224 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6v6t\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033329 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033329 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033286 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033449 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033426 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033504 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033557 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033507 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:43.033696 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033672 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:43.033766 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033690 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:18:43.033766 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:43.033880 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.033880 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.034001 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.033909 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.034142 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token\") pod \"9503baed-d359-4676-8faf-b191e6cd405a\" (UID: \"9503baed-d359-4676-8faf-b191e6cd405a\") " Aug 05 18:18:43.034298 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034280 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:43.034472 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034448 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.034538 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034474 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-credential-socket\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.034538 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034489 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9503baed-d359-4676-8faf-b191e6cd405a-istiod-ca-cert\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.034538 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034502 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-data\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.034538 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.034514 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-workload-certs\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.035809 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.035770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t" (OuterVolumeSpecName: "kube-api-access-h6v6t") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "kube-api-access-h6v6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:43.036103 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.036077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:18:43.036301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.036271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:18:43.036426 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.036399 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9503baed-d359-4676-8faf-b191e6cd405a" (UID: "9503baed-d359-4676-8faf-b191e6cd405a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:18:43.135259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.135223 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9503baed-d359-4676-8faf-b191e6cd405a-istio-podinfo\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.135259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.135251 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-istio-token\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.135259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.135261 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6v6t\" (UniqueName: \"kubernetes.io/projected/9503baed-d359-4676-8faf-b191e6cd405a-kube-api-access-h6v6t\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.135259 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.135269 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9503baed-d359-4676-8faf-b191e6cd405a-istio-envoy\") on node \"ip-10-0-138-228.ec2.internal\" DevicePath \"\"" Aug 05 18:18:43.811873 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.811844 2564 generic.go:356] "Generic (PLEG): container finished" podID="9503baed-d359-4676-8faf-b191e6cd405a" containerID="e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1" exitCode=0 Aug 05 18:18:43.812088 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.811894 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" Aug 05 18:18:43.812088 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.811936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerDied","Data":"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1"} Aug 05 18:18:43.812088 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.812013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp" event={"ID":"9503baed-d359-4676-8faf-b191e6cd405a","Type":"ContainerDied","Data":"dbd002bd39a90b0666b0d4953d95121a089f4cfae352552df4b0a95c179d32a4"} Aug 05 18:18:43.812088 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.812043 2564 scope.go:117] "RemoveContainer" containerID="e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1" Aug 05 18:18:43.820405 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.820386 2564 scope.go:117] "RemoveContainer" containerID="2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610" Aug 05 18:18:43.828431 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.828098 2564 scope.go:117] "RemoveContainer" containerID="e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1" Aug 05 18:18:43.828431 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:18:43.828400 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1\": container with ID starting with e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1 not found: ID does not exist" containerID="e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1" Aug 05 18:18:43.828602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.828442 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1"} err="failed to get container status \"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1\": rpc error: code = NotFound desc = could not find container \"e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1\": container with ID starting with e1ba1ddcd8502cee00a40cc75c0d12f6ccf99ba8cda060775ee243f5f2f7e9b1 not found: ID does not exist" Aug 05 18:18:43.828602 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.828462 2564 scope.go:117] "RemoveContainer" containerID="2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610" Aug 05 18:18:43.828805 ip-10-0-138-228 kubenswrapper[2564]: E0805 18:18:43.828777 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610\": container with ID starting with 2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610 not found: ID does not exist" containerID="2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610" Aug 05 18:18:43.828805 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.828804 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610"} err="failed to get container status \"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610\": rpc error: code = NotFound desc = could not find container \"2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610\": container with ID starting with 2b02ee0139550145cc940d72da26007d09ff54d3bd2c91503d6e29149d210610 not found: ID does not exist" Aug 05 18:18:43.831385 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.831365 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:18:43.835606 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:43.835585 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-3f8e0d-openshift-default-6b774f8b5b-584pp"] Aug 05 18:18:45.038873 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:45.038830 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9503baed-d359-4676-8faf-b191e6cd405a" path="/var/lib/kubelet/pods/9503baed-d359-4676-8faf-b191e6cd405a/volumes" Aug 05 18:18:48.695354 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:48.695323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:48.814653 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:48.814615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:48.924160 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:48.924126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:49.192211 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:49.192177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:55.298459 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:55.298422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:55.397410 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:55.397378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:55.552875 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:55.552783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:18:55.826371 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:18:55.826285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:19:02.140768 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:02.140735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pkcpp_d7999f09-dd22-470c-b935-36f29873537b/manager/0.log" Aug 05 18:19:02.372498 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:02.372447 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58c579b484-94lvx_b41391f9-a45c-4bb7-9c3f-d61787772f17/copy-manifests/0.log" Aug 05 18:19:02.378270 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:02.378248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58c579b484-94lvx_b41391f9-a45c-4bb7-9c3f-d61787772f17/manager/0.log" Aug 05 18:19:02.998857 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:02.998828 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29765865-7d9tb_83c0dfd6-7c12-4c0d-b52a-ec499b990280/cleanup/0.log" Aug 05 18:19:05.224368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:05.224337 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg_3c56e1ac-48d2-4936-92e8-32e8fcfcf61b/istio-proxy/0.log" Aug 05 18:19:05.341396 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:05.341367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zjc2h_1376d3c1-28e8-40e6-91d6-cf88e69f6327/discovery/0.log" Aug 05 18:19:05.688178 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:05.688147 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/1.log" Aug 05 18:19:06.056383 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:06.056355 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d8d8f5b-rrnmh_6462a8f5-6939-4711-9294-168ea9216841/router/0.log" Aug 05 18:19:06.403238 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:06.403163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9l9ck_5910bc30-655c-42ee-8133-f1b215f3a0f2/main/0.log" Aug 05 18:19:06.634967 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:06.634921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ltqdb_e3d163fe-9c4b-4fcb-9ce6-95beaa988f65/main/0.log" Aug 05 18:19:06.753311 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:06.753236 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhfhxt_ef607391-12a1-40fc-8fa0-10bfb5bfb38e/main/0.log" Aug 05 18:19:12.856416 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:12.856370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:19:12.955035 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:12.955001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" limit=1024 Aug 05 18:19:15.659704 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:15.659674 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vb66t_8246d4ad-44f1-4c46-8f31-62348f5a71d3/global-pull-secret-syncer/0.log" Aug 05 18:19:15.709461 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:15.709431 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2jfxr_b936f9c2-6a92-4cfa-8130-8c9573c4b58f/konnectivity-agent/0.log" Aug 05 18:19:15.827466 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:15.827435 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-228.ec2.internal_1cbf03bc1f47a41c1d8a87dcec2c129b/haproxy/0.log" Aug 05 18:19:23.049430 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.049402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-p8pcq_e1e319f2-7d5e-4991-9c24-713a8d52ce1d/cluster-monitoring-operator/0.log" Aug 05 18:19:23.218011 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.217917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-s87fv_7de704b1-f978-4db8-9043-9e37630d4aa4/monitoring-plugin/0.log" Aug 05 18:19:23.334550 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.334527 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqwzl_11f51db9-c061-4439-b97a-6dc61e8f53b7/node-exporter/0.log" Aug 05 18:19:23.358151 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.358122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqwzl_11f51db9-c061-4439-b97a-6dc61e8f53b7/kube-rbac-proxy/0.log" Aug 05 18:19:23.380290 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.380264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqwzl_11f51db9-c061-4439-b97a-6dc61e8f53b7/init-textfile/0.log" Aug 05 18:19:23.610758 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.610716 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/prometheus/0.log" Aug 05 18:19:23.633377 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.633349 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/config-reloader/0.log" Aug 05 18:19:23.659038 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.659013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/thanos-sidecar/0.log" Aug 05 18:19:23.685488 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.685466 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/kube-rbac-proxy-web/0.log" Aug 05 18:19:23.708967 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.708917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/kube-rbac-proxy/0.log" Aug 05 18:19:23.731553 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.731528 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/kube-rbac-proxy-thanos/0.log" Aug 05 18:19:23.755374 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.755357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0f9d50b-d5fb-44cc-a977-c8bf05a0fe41/init-config-reloader/0.log" Aug 05 18:19:23.784948 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.784925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-wrc2l_d723d786-ee1b-44d9-b406-35a7b61e9bb6/prometheus-operator/0.log" Aug 05 18:19:23.805747 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:23.805727 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-wrc2l_d723d786-ee1b-44d9-b406-35a7b61e9bb6/kube-rbac-proxy/0.log" Aug 05 18:19:26.213368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213339 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg"] Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213654 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213664 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213676 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213682 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213688 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213694 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213700 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213706 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213715 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213720 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213725 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213730 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.213736 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213742 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213748 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213754 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213760 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213807 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e576beb-e550-4696-bc4b-c948e6ea6f6a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213815 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213821 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3794d9-5d3e-4f6f-9535-7ceca6dd6b37" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213827 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9503baed-d359-4676-8faf-b191e6cd405a" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213833 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.214140 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.213838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a950a493-0bda-4cfd-8979-2e6caf43a257" containerName="istio-proxy" Aug 05 18:19:26.216494 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.216473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.219075 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.219051 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f9829\"/\"openshift-service-ca.crt\"" Aug 05 18:19:26.219213 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.219058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f9829\"/\"default-dockercfg-6655g\"" Aug 05 18:19:26.220116 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.220094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f9829\"/\"kube-root-ca.crt\"" Aug 05 18:19:26.228894 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.228856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg"] Aug 05 18:19:26.229741 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.229714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-podres\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.229852 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.229769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-proc\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.229912 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.229850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-sys\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.229912 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.229884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-lib-modules\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.230043 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.229942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnltn\" (UniqueName: \"kubernetes.io/projected/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-kube-api-access-pnltn\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.330795 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-podres\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.330795 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-proc\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-sys\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-lib-modules\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnltn\" (UniqueName: \"kubernetes.io/projected/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-kube-api-access-pnltn\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-podres\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-sys\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.330944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-proc\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.331033 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.331027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-lib-modules\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.340273 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.340251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnltn\" (UniqueName: \"kubernetes.io/projected/fa8192be-9a41-4ca4-89bb-d4d59588b4f5-kube-api-access-pnltn\") pod \"perf-node-gather-daemonset-nn8mg\" (UID: \"fa8192be-9a41-4ca4-89bb-d4d59588b4f5\") " pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.525920 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.525817 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:26.649080 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.649052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-l9kqb_b64bd4b2-fbd2-41c2-beb8-9da5351adb1f/download-server/0.log" Aug 05 18:19:26.654990 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:26.654948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg"] Aug 05 18:19:26.657442 ip-10-0-138-228 kubenswrapper[2564]: W0805 18:19:26.657418 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa8192be_9a41_4ca4_89bb_d4d59588b4f5.slice/crio-38e1c2b4ab5efbc3122ea6b6a35357f4600a787d243683c6efca61a0e562b7f6 WatchSource:0}: Error finding container 38e1c2b4ab5efbc3122ea6b6a35357f4600a787d243683c6efca61a0e562b7f6: Status 404 returned error can't find the container with id 38e1c2b4ab5efbc3122ea6b6a35357f4600a787d243683c6efca61a0e562b7f6 Aug 05 18:19:27.004138 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:27.004105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" event={"ID":"fa8192be-9a41-4ca4-89bb-d4d59588b4f5","Type":"ContainerStarted","Data":"6bd6f8df3626ca9b9cfb2e2b5be611ca4ec3c567ca878b2085c03ace51d35b25"} Aug 05 18:19:27.004341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:27.004153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" event={"ID":"fa8192be-9a41-4ca4-89bb-d4d59588b4f5","Type":"ContainerStarted","Data":"38e1c2b4ab5efbc3122ea6b6a35357f4600a787d243683c6efca61a0e562b7f6"} Aug 05 18:19:27.004341 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:27.004210 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:27.021116 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:27.021070 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" podStartSLOduration=1.021056669 podStartE2EDuration="1.021056669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 18:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:19:27.020332003 +0000 UTC m=+3600.451478254" watchObservedRunningTime="2026-08-05 18:19:27.021056669 +0000 UTC m=+3600.452202960" Aug 05 18:19:28.436095 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:28.436060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vnsvp_4367aa03-3c82-49a8-8111-a9f0cd2e124f/dns/0.log" Aug 05 18:19:28.457001 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:28.456980 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vnsvp_4367aa03-3c82-49a8-8111-a9f0cd2e124f/kube-rbac-proxy/0.log" Aug 05 18:19:28.528519 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:28.528485 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9rcr_e6d3d3d4-1f92-48ad-8aeb-7bea6f342378/dns-node-resolver/0.log" Aug 05 18:19:29.079490 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:29.079457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dd84f8cf4-r96wh_1130020e-8f63-445f-9ba9-eb50c9c91b10/registry/0.log" Aug 05 18:19:29.149001 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:29.148972 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8rkq_4e67ba62-36b0-4223-83f8-00eb8fbc53b9/node-ca/0.log" Aug 05 18:19:30.073432 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:30.073404 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg_3c56e1ac-48d2-4936-92e8-32e8fcfcf61b/istio-proxy/0.log" Aug 05 18:19:30.234905 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:30.234872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zjc2h_1376d3c1-28e8-40e6-91d6-cf88e69f6327/discovery/0.log" Aug 05 18:19:30.308951 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:30.308920 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/0.log" Aug 05 18:19:30.316039 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:30.316017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/1.log" Aug 05 18:19:30.418368 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:30.418289 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d8d8f5b-rrnmh_6462a8f5-6939-4711-9294-168ea9216841/router/0.log" Aug 05 18:19:31.244899 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:31.244870 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6xz5_8927124e-9aee-4387-b5f1-cbcd8b6782b4/serve-healthcheck-canary/0.log" Aug 05 18:19:31.940261 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:31.940228 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrl49_8f8ff391-db3d-431f-bf38-be70bd1394ba/kube-rbac-proxy/0.log" Aug 05 18:19:31.960927 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:31.960905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrl49_8f8ff391-db3d-431f-bf38-be70bd1394ba/exporter/0.log" Aug 05 18:19:31.988013 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:31.987988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrl49_8f8ff391-db3d-431f-bf38-be70bd1394ba/extractor/0.log" Aug 05 18:19:33.018657 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:33.018632 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f9829/perf-node-gather-daemonset-nn8mg" Aug 05 18:19:34.860126 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:34.860095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pkcpp_d7999f09-dd22-470c-b935-36f29873537b/manager/0.log" Aug 05 18:19:34.940021 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:34.939986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58c579b484-94lvx_b41391f9-a45c-4bb7-9c3f-d61787772f17/manager/0.log" Aug 05 18:19:34.960324 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:34.960303 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58c579b484-94lvx_b41391f9-a45c-4bb7-9c3f-d61787772f17/copy-manifests/0.log" Aug 05 18:19:36.503393 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.503312 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57fb49d769-6mbqf_a0a7b9b1-d0e1-4f82-adad-664f44a3753b/manager/0.log" Aug 05 18:19:36.706516 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.706477 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg_3c56e1ac-48d2-4936-92e8-32e8fcfcf61b/istio-proxy/0.log" Aug 05 18:19:36.861569 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.861536 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zjc2h_1376d3c1-28e8-40e6-91d6-cf88e69f6327/discovery/0.log" Aug 05 18:19:36.913610 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.913583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/0.log" Aug 05 18:19:36.919972 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.919935 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/1.log" Aug 05 18:19:36.999491 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:36.999462 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d8d8f5b-rrnmh_6462a8f5-6939-4711-9294-168ea9216841/router/0.log" Aug 05 18:19:38.419180 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:38.419152 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n2qlg_3c56e1ac-48d2-4936-92e8-32e8fcfcf61b/istio-proxy/0.log" Aug 05 18:19:38.571597 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:38.571566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zjc2h_1376d3c1-28e8-40e6-91d6-cf88e69f6327/discovery/0.log" Aug 05 18:19:38.621639 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:38.621603 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/0.log" Aug 05 18:19:38.628062 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:38.628039 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p2lzb_7ef34bc7-0b58-4eca-88f4-52fcad58714d/istio-proxy/1.log" Aug 05 18:19:38.710301 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:38.710218 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d8d8f5b-rrnmh_6462a8f5-6939-4711-9294-168ea9216841/router/0.log" Aug 05 18:19:43.108206 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.108180 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/kube-multus-additional-cni-plugins/0.log" Aug 05 18:19:43.131898 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.131871 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/egress-router-binary-copy/0.log" Aug 05 18:19:43.153609 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.153590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/cni-plugins/0.log" Aug 05 18:19:43.174297 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.174279 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/bond-cni-plugin/0.log" Aug 05 18:19:43.196965 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.196943 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/routeoverride-cni/0.log" Aug 05 18:19:43.221479 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.221460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/whereabouts-cni-bincopy/0.log" Aug 05 18:19:43.242690 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.242669 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8gsgs_aaf06d87-8c29-4f6d-8c51-0d9834cab5ea/whereabouts-cni/0.log" Aug 05 18:19:43.623307 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.623279 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pklxj_d793d02b-2909-4792-a697-cf7b93a36426/kube-multus/0.log" Aug 05 18:19:43.643015 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.642990 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2hc9s_a3311739-a779-4c3d-a848-5482e3cee472/network-metrics-daemon/0.log" Aug 05 18:19:43.661670 ip-10-0-138-228 kubenswrapper[2564]: I0805 18:19:43.661641 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2hc9s_a3311739-a779-4c3d-a848-5482e3cee472/kube-rbac-proxy/0.log"