Jul 29 15:44:33.934506 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 15:44:33.934518 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 15:44:33.934527 ip-10-0-130-153 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 15:44:33.934883 ip-10-0-130-153 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 15:44:44.046598 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 15:44:44.046621 ip-10-0-130-153 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c5fb2e485794492fbe1c083fb02c0ffb -- Jul 29 15:46:42.592822 ip-10-0-130-153 systemd[1]: Starting Kubernetes Kubelet... Jul 29 15:46:43.033221 ip-10-0-130-153 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:43.033221 ip-10-0-130-153 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 15:46:43.033221 ip-10-0-130-153 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:43.033221 ip-10-0-130-153 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 15:46:43.033221 ip-10-0-130-153 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 15:46:43.036097 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.035990 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 15:46:43.042655 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042639 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.042655 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042656 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042660 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042663 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042667 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042670 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042673 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042676 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042679 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042681 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042684 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042687 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042689 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042692 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042696 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042712 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042715 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042718 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042721 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042724 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042727 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.042726 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042729 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042732 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042735 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042738 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042741 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042744 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042746 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042749 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042752 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042754 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042757 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042759 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042762 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042764 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042767 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042769 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042772 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042775 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042777 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042780 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.043248 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042783 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042785 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042788 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042792 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042796 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042809 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042813 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042815 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042818 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042821 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042824 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042827 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042830 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042834 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042839 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042842 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042845 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042847 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042851 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.043832 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042853 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042856 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042859 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042861 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042864 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042869 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042873 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042875 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042878 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042881 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042884 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042886 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042889 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042892 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042895 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042897 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042900 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042903 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042905 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.044376 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042945 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042994 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.042999 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043004 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043009 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043014 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043018 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043023 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043029 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043034 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043038 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043043 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043048 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043701 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043714 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043718 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043723 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043728 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043732 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.043737 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.044854 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044790 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044801 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044805 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044809 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044812 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044815 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044818 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044821 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044824 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044826 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044829 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044832 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044835 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044838 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044841 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044844 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044846 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044849 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044853 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044857 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.045345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044861 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044864 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044867 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044870 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044873 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044875 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044878 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044880 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044883 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044885 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044888 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044890 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044893 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044896 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044899 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044902 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044904 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044907 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044910 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044912 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.045867 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044915 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044918 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044921 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044923 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044926 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044929 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044931 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044934 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044937 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044942 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044944 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044947 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044951 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044954 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044957 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044960 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044963 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044966 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044968 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.046364 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044971 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044974 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044976 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044979 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044982 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044986 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044989 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044994 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044997 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.044999 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045002 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045004 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045007 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045009 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045012 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045014 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045017 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045020 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045023 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.046855 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045025 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045028 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045030 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045033 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045036 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045038 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045041 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045043 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045046 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045048 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045051 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045053 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045056 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045058 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045060 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045063 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045065 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045068 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045072 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045074 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.047319 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045078 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045502 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045508 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045511 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045515 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045518 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045520 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045523 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045526 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045529 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045532 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045534 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045537 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045540 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045543 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045545 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045548 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045550 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045553 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.047818 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045556 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045559 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045561 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045564 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045566 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045568 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045571 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045574 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045576 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045579 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045581 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045584 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045587 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045590 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045592 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045595 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045598 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045600 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045602 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045605 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.048280 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045607 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045610 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045612 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045615 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045618 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045620 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045622 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045625 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045627 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045630 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045632 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045635 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045638 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045640 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045643 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045645 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045648 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045651 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045654 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045656 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.048774 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045659 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045662 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045664 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045666 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045670 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045673 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045677 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045680 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045683 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045686 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045688 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045691 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045694 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045696 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045699 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045702 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045705 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045709 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045713 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.049291 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045715 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045718 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045720 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045723 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045726 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045728 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045730 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045733 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045735 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045738 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045742 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045745 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045748 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045753 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045756 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045759 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045761 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045764 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045767 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.049784 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045771 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045773 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.045776 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045847 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045858 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045863 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045868 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045872 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045875 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045880 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045884 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045887 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045890 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045893 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045896 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045899 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045902 2574 flags.go:64] FLAG: --cgroup-root="" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045905 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045908 2574 flags.go:64] FLAG: --client-ca-file="" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045911 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045913 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045918 2574 flags.go:64] FLAG: --cluster-domain="" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045921 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 15:46:43.050242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045924 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045926 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045929 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045935 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045938 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045941 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045944 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045947 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045950 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045953 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045956 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045959 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045965 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045968 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045971 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045974 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045977 2574 flags.go:64] FLAG: --enable-server="true" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045979 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045983 2574 flags.go:64] FLAG: --event-burst="100" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045986 2574 flags.go:64] FLAG: --event-qps="50" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045989 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045991 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045994 2574 flags.go:64] FLAG: --eviction-hard="" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.045997 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 15:46:43.050834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046000 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046003 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046008 2574 flags.go:64] FLAG: --eviction-soft="" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046011 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046013 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046016 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046019 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046022 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046025 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046027 2574 flags.go:64] FLAG: --feature-gates="" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046031 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046035 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046038 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046041 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046044 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046047 2574 flags.go:64] FLAG: --help="false" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046050 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-153.ec2.internal" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046053 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046056 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046059 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046063 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046066 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046069 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 15:46:43.051397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046072 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046074 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046077 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046080 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046083 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046086 2574 flags.go:64] FLAG: --kube-reserved="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046089 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046091 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046094 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046097 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046100 2574 flags.go:64] FLAG: --lock-file="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046104 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046107 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046109 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046115 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046118 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046121 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046124 2574 flags.go:64] FLAG: --logging-format="text" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046127 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046130 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046134 2574 flags.go:64] FLAG: --manifest-url="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046136 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046141 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046143 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046147 2574 flags.go:64] FLAG: --max-pods="110" Jul 29 15:46:43.052015 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046150 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046153 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046156 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046159 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046162 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046165 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046168 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046174 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046177 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046180 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046183 2574 flags.go:64] FLAG: --pod-cidr="" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046186 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046191 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046194 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046197 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046200 2574 flags.go:64] FLAG: --port="10250" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046203 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046206 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f913d7e0fa2ec04" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046209 2574 flags.go:64] FLAG: --qos-reserved="" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046213 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046216 2574 flags.go:64] FLAG: --register-node="true" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046219 2574 flags.go:64] FLAG: --register-with-taints="" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046222 2574 flags.go:64] FLAG: --registry-burst="10" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046225 2574 flags.go:64] FLAG: --registry-qps="5" Jul 29 15:46:43.052772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046228 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046231 2574 flags.go:64] FLAG: --reserved-memory="" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046234 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046241 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046244 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046246 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046249 2574 flags.go:64] FLAG: --runonce="false" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046252 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046255 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046258 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046260 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046263 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046267 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046270 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046273 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046276 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046279 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046282 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046285 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046288 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046291 2574 flags.go:64] FLAG: --system-cgroups="" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046293 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046299 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046301 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046304 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 15:46:43.053351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046308 2574 flags.go:64] FLAG: --tls-min-version="" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046311 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046315 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046318 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046321 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046323 2574 flags.go:64] FLAG: --v="2" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046327 2574 flags.go:64] FLAG: --version="false" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046331 2574 flags.go:64] FLAG: --vmodule="" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046334 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.046337 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046459 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046464 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046467 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046470 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046472 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046475 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046477 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046493 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046496 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046499 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046502 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046505 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.053980 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046507 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046510 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046513 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046516 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046518 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046521 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046523 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046526 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046528 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046531 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046533 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046536 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046540 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046542 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046545 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046548 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046550 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046553 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046555 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046558 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.054625 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046562 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046565 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046568 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046570 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046573 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046575 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046578 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046581 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046583 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046586 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046588 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046591 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046594 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046597 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046599 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046602 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046604 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046607 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046609 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.055116 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046612 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046614 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046617 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046619 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046622 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046627 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046629 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046632 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046636 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046639 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046642 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046645 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046648 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046651 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046654 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046657 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046659 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046663 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046667 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046670 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.055597 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046673 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046676 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046679 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046682 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046685 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046688 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046692 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046695 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046697 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046700 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046703 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046707 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046710 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046713 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046716 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046719 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046722 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046726 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046729 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.056084 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046731 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046734 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046737 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046739 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046741 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046744 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046748 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046750 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.046753 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.047911 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.055355 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.055370 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.055388 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.055394 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055451 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.056571 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055456 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055461 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055465 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055469 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055472 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055475 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055493 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055498 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055501 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055504 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055507 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055509 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055512 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055514 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055517 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055520 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055522 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055525 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.056941 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055527 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055530 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055532 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055535 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055538 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055540 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055543 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055545 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055548 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055550 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055554 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055557 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055559 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055562 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055566 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055570 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055573 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055575 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055586 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.057385 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055589 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055592 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055595 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055597 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055600 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055603 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055606 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055609 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055612 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055615 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055618 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055620 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055623 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055625 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055628 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055630 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055633 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055635 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055638 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055640 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.057870 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055643 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055645 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055648 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055651 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055654 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055656 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055659 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055662 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055664 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055667 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055670 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055672 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055675 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055678 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055680 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055682 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055685 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055688 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055690 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055693 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.058378 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055695 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055698 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055700 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055703 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055705 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055708 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055710 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055713 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055715 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055718 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055720 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055722 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055725 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055727 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055730 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055734 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055736 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055739 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055742 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055744 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.058950 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055747 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.055752 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055887 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055891 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055894 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055898 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055901 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055904 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055907 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055910 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055912 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055915 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055917 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055920 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055922 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055925 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 15:46:43.059442 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055928 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055930 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055933 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055935 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055937 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055940 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055942 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055945 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055947 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055950 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055952 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055955 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055958 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055961 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055963 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055966 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055969 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055972 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055975 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 15:46:43.059842 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055978 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055980 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055983 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055986 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055988 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055991 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055993 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055996 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.055999 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056001 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056004 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056007 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056009 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056012 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056014 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056017 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056019 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056022 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056024 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 15:46:43.060322 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056026 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056029 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056031 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056034 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056037 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056039 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056042 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056045 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056047 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056050 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056052 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056056 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056059 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056062 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056065 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056067 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056070 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056073 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056076 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056078 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 15:46:43.060802 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056081 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056083 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056086 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056088 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056091 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056093 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056096 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056099 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056101 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056103 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056106 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056108 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056111 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056113 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056117 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056120 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056123 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056126 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056129 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 15:46:43.061279 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056132 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056134 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056137 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056139 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056142 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056145 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056147 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:43.056150 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.056155 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.056249 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.058345 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.059706 2574 server.go:1019] "Starting client certificate rotation" Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.059817 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 15:46:43.061787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.059863 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 15:46:43.062133 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.062118 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 15:46:43.064001 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.063980 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 15:46:43.073958 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.073942 2574 log.go:25] "Validated CRI v1 runtime API" Jul 29 15:46:43.077248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.077232 2574 log.go:25] "Validated CRI v1 image API" Jul 29 15:46:43.080012 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.079995 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 15:46:43.082513 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.082470 2574 fs.go:135] Filesystem UUIDs: map[0d5df566-a1cb-40ee-b3f4-e1cac5648191:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b08abebf-0c49-4f7d-a3f9-670da7617ec3:/dev/nvme0n1p3] Jul 29 15:46:43.082574 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.082513 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 15:46:43.087584 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.087460 2574 manager.go:216] Machine: {Timestamp:2026-07-29 15:46:43.086978221 +0000 UTC m=+0.368031002 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100362 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a505d7a5d24642bd71dfafe7feca SystemUUID:ec29a505-d7a5-d246-42bd-71dfafe7feca BootID:c5fb2e48-5794-492f-be1c-083fb02c0ffb Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:59:c4:29:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:59:c4:29:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:d0:8d:5d:22:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 15:46:43.087584 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.087580 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 15:46:43.087688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.087654 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 15:46:43.090035 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.090019 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 15:46:43.090470 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.090449 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 15:46:43.090627 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.090474 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 15:46:43.091239 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.091229 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 15:46:43.091270 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.091240 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 15:46:43.091270 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.091249 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 15:46:43.091270 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.091259 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 15:46:43.092276 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.092266 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 15:46:43.094088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094077 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 29 15:46:43.094232 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094223 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 15:46:43.094266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094252 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 29 15:46:43.094266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094262 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 15:46:43.094322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094280 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 15:46:43.094322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094290 2574 kubelet.go:398] "Adding apiserver pod source" Jul 29 15:46:43.094322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094298 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 15:46:43.094681 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094663 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 15:46:43.094779 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.094691 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 15:46:43.097269 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.097252 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 15:46:43.099155 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.099142 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 15:46:43.099205 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.099172 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 15:46:43.102078 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.102053 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 15:46:43.102172 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102158 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 15:46:43.102218 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102186 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 15:46:43.102218 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102193 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 15:46:43.102218 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102199 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 15:46:43.102218 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102208 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102244 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.102134 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102254 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102261 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102268 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102273 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102281 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 15:46:43.102326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.102287 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 15:46:43.103097 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.103086 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 15:46:43.103130 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.103098 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 15:46:43.106107 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.106094 2574 server.go:1268] "Started kubelet" Jul 29 15:46:43.106172 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.106090 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-153.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 15:46:43.106221 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.106164 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 15:46:43.106266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.106218 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 15:46:43.106455 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.106362 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 15:46:43.107014 ip-10-0-130-153 systemd[1]: Started Kubernetes Kubelet. Jul 29 15:46:43.108288 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.108270 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 15:46:43.110858 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.110831 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rj962" Jul 29 15:46:43.111345 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.111331 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 29 15:46:43.112137 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.112117 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 15:46:43.117335 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.116384 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-153.ec2.internal.18c6cd46106e9bc9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-153.ec2.internal,UID:ip-10-0-130-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-153.ec2.internal,},FirstTimestamp:2026-07-29 15:46:43.106069449 +0000 UTC m=+0.387122232,LastTimestamp:2026-07-29 15:46:43.106069449 +0000 UTC m=+0.387122232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-153.ec2.internal,}" Jul 29 15:46:43.117717 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.117703 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 15:46:43.117717 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.117712 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 15:46:43.117819 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.117805 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:43.118302 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118288 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 15:46:43.118413 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118396 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 15:46:43.118531 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118290 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 15:46:43.118615 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118598 2574 factory.go:55] Registering systemd factory Jul 29 15:46:43.118683 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118641 2574 factory.go:223] Registration of the systemd container factory successfully Jul 29 15:46:43.119088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118977 2574 factory.go:153] Registering CRI-O factory Jul 29 15:46:43.119088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.118989 2574 factory.go:223] Registration of the crio container factory successfully Jul 29 15:46:43.119088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119032 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 15:46:43.119088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119054 2574 factory.go:103] Registering Raw factory Jul 29 15:46:43.119088 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119070 2574 manager.go:1196] Started watching for new ooms in manager Jul 29 15:46:43.119385 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119358 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 29 15:46:43.119461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119447 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 29 15:46:43.119461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119454 2574 manager.go:349] Starting recovery of all containers Jul 29 15:46:43.119625 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.119598 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rj962" Jul 29 15:46:43.128002 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.127982 2574 manager.go:354] Recovery completed Jul 29 15:46:43.128615 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.128549 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:43.130065 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.130044 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 29 15:46:43.131051 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:43.131032 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-153.ec2.internal\" not found" node="ip-10-0-130-153.ec2.internal" Jul 29 15:46:43.132943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.132932 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:43.134651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.134567 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:43.134651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.134601 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:43.134651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.134616 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:43.135135 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135120 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 15:46:43.135135 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135132 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 15:46:43.135135 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135136 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:43.135294 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135147 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 29 15:46:43.135796 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135781 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:43.135872 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135811 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:43.135872 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.135825 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:43.136599 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.136573 2574 policy_none.go:49] "None policy: Start" Jul 29 15:46:43.136637 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.136602 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 15:46:43.136637 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.136612 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 15:46:43.137691 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:43.137680 2574 policy_none.go:47] "Start" Jul 29 15:46:44.221689 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.221530 2574 manager.go:337] "Starting Device Plugin manager" Jul 29 15:46:44.221796 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.221714 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 15:46:44.221796 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.221727 2574 server.go:87] "Starting device plugin registration server" Jul 29 15:46:44.222009 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.221997 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 15:46:44.222049 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.222014 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 15:46:44.222136 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.222120 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 15:46:44.222227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.222211 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 15:46:44.222227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.222225 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 15:46:44.222736 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.222714 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 15:46:44.222795 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.222774 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.322488 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.322449 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:44.323831 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.323415 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:44.323831 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.323455 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:44.323831 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.323472 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:44.323831 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.323511 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.325473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.325449 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 15:46:44.326699 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.326647 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 15:46:44.326699 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.326693 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 15:46:44.326830 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.326717 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 15:46:44.326830 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.326757 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 15:46:44.328301 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.328279 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:44.331520 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.331506 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.331593 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.331527 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-153.ec2.internal\": node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.380505 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.380480 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.427611 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.427579 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"] Jul 29 15:46:44.427691 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.427657 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:44.428575 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.428559 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:44.428650 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.428588 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:44.428650 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.428598 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:44.429894 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.429881 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:44.430034 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.430074 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430052 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:44.430571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430557 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:44.430638 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430584 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:44.430638 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430594 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:44.430638 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430557 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:44.430727 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430660 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:44.430727 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.430673 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:44.431795 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.431780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.431837 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.431810 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 15:46:44.432473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.432457 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory" Jul 29 15:46:44.432548 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.432488 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 15:46:44.432548 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.432503 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID" Jul 29 15:46:44.454683 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.454662 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-153.ec2.internal\" not found" node="ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.459057 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.459044 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-153.ec2.internal\" not found" node="ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.463204 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.463189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.463253 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.463213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.463253 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.463231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7c1d29852edb376853ce02192f5c250-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"c7c1d29852edb376853ce02192f5c250\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.481212 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.481194 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.563554 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7c1d29852edb376853ce02192f5c250-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"c7c1d29852edb376853ce02192f5c250\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.563554 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.563695 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.563695 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.563695 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06fba2a84e38e9d2ca29be30525318bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"06fba2a84e38e9d2ca29be30525318bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.563695 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.563620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7c1d29852edb376853ce02192f5c250-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"c7c1d29852edb376853ce02192f5c250\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.581325 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.581276 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.682092 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.682052 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.757274 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.757248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.762244 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:44.762220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:44.782545 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.782522 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.883183 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.883084 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:44.983584 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:44.983540 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found" Jul 29 15:46:45.007108 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.007084 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:45.061075 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.061045 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" Jul 29 15:46:45.077826 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.077805 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 15:46:45.079898 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.079881 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" Jul 29 15:46:45.087097 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.087079 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 15:46:45.102164 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.102145 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 15:46:45.102290 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.102273 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:45.102343 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.102273 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:45.102343 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.102276 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 15:46:45.137458 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.137432 2574 apiserver.go:52] "Watching apiserver" Jul 29 15:46:45.157174 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.157145 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 15:46:45.157503 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.157479 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kzhbk","kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw","openshift-cluster-node-tuning-operator/tuned-9fs2g","openshift-image-registry/node-ca-5ptlz","openshift-multus/multus-vx7tl","openshift-network-diagnostics/network-check-target-7224n","kube-system/global-pull-secret-syncer-w9k2g","openshift-dns/node-resolver-994kz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal","openshift-multus/multus-additional-cni-plugins-s5cv4","openshift-multus/network-metrics-daemon-jmxs9","openshift-network-operator/iptables-alerter-26n6r","openshift-ovn-kubernetes/ovnkube-node-r4hgq"] Jul 29 15:46:45.159187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.159168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:45.159293 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.159227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:45.160356 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.160332 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 15:46:45.160470 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.160417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.161818 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.161801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.162813 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.162790 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.162911 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.162820 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 15:46:45.163003 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.162916 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.163060 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.163040 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dsstd\"" Jul 29 15:46:45.163546 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.163507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.166111 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.164281 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.166111 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.164282 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 15:46:45.166111 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.164676 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.166111 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.164742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2xh6\"" Jul 29 15:46:45.166346 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-device-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166346 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-sys-fs\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166433 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjztg\" (UniqueName: \"kubernetes.io/projected/420ce120-0f79-4acc-89cd-a53bdf44543d-kube-api-access-bjztg\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166433 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:45.166433 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.166540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-socket-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-registration-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.166688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0463fa5f-4621-4e6a-893e-5df125d67cb7-iptables-alerter-script\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.166688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0463fa5f-4621-4e6a-893e-5df125d67cb7-host-slash\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.166688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4dj\" (UniqueName: \"kubernetes.io/projected/0463fa5f-4621-4e6a-893e-5df125d67cb7-kube-api-access-zc4dj\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.166874 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166660 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 15:41:44 +0000 UTC" deadline="2028-03-06 03:50:03.882208195 +0000 UTC" Jul 29 15:46:45.166874 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.166721 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14052h3m18.715491434s" Jul 29 15:46:45.167344 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.167325 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.167423 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.167330 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.167484 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.167402 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q8279\"" Jul 29 15:46:45.168260 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.168236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.169523 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.169505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.170536 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.170515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 15:46:45.170616 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.170572 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.170884 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.170870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.171322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.171305 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.172174 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.172159 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6q7s6\"" Jul 29 15:46:45.172248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.172201 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 15:46:45.172310 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.172299 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.172432 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.172416 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.173301 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.173283 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.173435 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.173420 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b5j8b\"" Jul 29 15:46:45.173753 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.173739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.173827 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.173791 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:45.173883 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.173834 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 15:46:45.174788 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.174770 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.175060 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.175041 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j7lrt\"" Jul 29 15:46:45.175156 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.175080 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 15:46:45.175156 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.175089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8dnqm\"" Jul 29 15:46:45.175339 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.175320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.175940 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.175921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 15:46:45.176740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176296 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 15:46:45.176740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176357 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 15:46:45.176740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176368 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.176740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176373 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 15:46:45.176740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176575 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 15:46:45.176966 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.176835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.176966 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.176891 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:45.177723 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.177704 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 15:46:45.177723 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.177719 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-blxnj\"" Jul 29 15:46:45.177823 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.177762 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 15:46:45.178850 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.178835 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 15:46:45.178961 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.178943 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6nln2\"" Jul 29 15:46:45.179078 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.178971 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 15:46:45.179190 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.179172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 15:46:45.183312 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.183292 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:45.214394 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.214372 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhl55" Jul 29 15:46:45.222513 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.222494 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhl55" Jul 29 15:46:45.233900 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.233864 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c1d29852edb376853ce02192f5c250.slice/crio-b3a548c8eeba8399aaee20e282f3f912c914b166f51b514d57cb7b8d3c6fee95 WatchSource:0}: Error finding container b3a548c8eeba8399aaee20e282f3f912c914b166f51b514d57cb7b8d3c6fee95: Status 404 returned error can't find the container with id b3a548c8eeba8399aaee20e282f3f912c914b166f51b514d57cb7b8d3c6fee95 Jul 29 15:46:45.240419 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.240400 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:46:45.261777 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.261761 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 15:46:45.267006 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.266970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-serviceca\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-bin\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-script-lib\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-dbus\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-modprobe-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysconfig\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.267132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-sys-fs\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-netns\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-sys-fs\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-etc-kubernetes\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqbp\" (UniqueName: \"kubernetes.io/projected/af4961fd-5883-4b04-8eb0-1c950898d3ab-kube-api-access-vlqbp\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-cnibin\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-os-release\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-config\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-os-release\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.267449 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63d1846d-8f08-4e0e-bd53-4670efb97c85-konnectivity-ca\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4dj\" (UniqueName: \"kubernetes.io/projected/0463fa5f-4621-4e6a-893e-5df125d67cb7-kube-api-access-zc4dj\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-kubernetes\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-system-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-multus-certs\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57zs\" (UniqueName: \"kubernetes.io/projected/98be5804-426c-4fa2-883c-0e9e0208cbd4-kube-api-access-b57zs\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-lib-modules\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-tuned\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-netns\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-kubelet\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-host\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bb48\" (UniqueName: \"kubernetes.io/projected/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-kube-api-access-4bb48\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-systemd-units\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.267879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-etc-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-log-socket\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-conf\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-slash\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.267954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-systemd\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-run\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-sys\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-k8s-cni-cncf-io\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-tmp\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268093 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4961fd-5883-4b04-8eb0-1c950898d3ab-tmp-dir\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-socket-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-ovn\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-host\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-cnibin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.268406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-multus\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4961fd-5883-4b04-8eb0-1c950898d3ab-hosts-file\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-daemon-config\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-socket-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-system-cni-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-systemd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-node-log\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-cni-binary-copy\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-socket-dir-parent\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-bin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-var-lib-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkc4\" (UniqueName: \"kubernetes.io/projected/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-kube-api-access-jxkc4\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.269094 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-conf-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrct\" (UniqueName: \"kubernetes.io/projected/40634acf-43ac-4123-81b1-9136df1c4d6b-kube-api-access-6qrct\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63d1846d-8f08-4e0e-bd53-4670efb97c85-agent-certs\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-var-lib-kubelet\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-device-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjztg\" (UniqueName: \"kubernetes.io/projected/420ce120-0f79-4acc-89cd-a53bdf44543d-kube-api-access-bjztg\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-env-overrides\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-kubelet-config\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-hostroot\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-device-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-registration-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpc6l\" (UniqueName: \"kubernetes.io/projected/648d11ed-cc61-426a-913f-cde108d6b63c-kube-api-access-cpc6l\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.269569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/420ce120-0f79-4acc-89cd-a53bdf44543d-registration-dir\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0463fa5f-4621-4e6a-893e-5df125d67cb7-iptables-alerter-script\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0463fa5f-4621-4e6a-893e-5df125d67cb7-host-slash\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-kubelet\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.268972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-netd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.269011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0463fa5f-4621-4e6a-893e-5df125d67cb7-host-slash\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.269018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovn-node-metrics-cert\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.269040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kglw\" (UniqueName: \"kubernetes.io/projected/839cb74e-cea4-4bfb-99fb-010af6495d0b-kube-api-access-8kglw\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.270005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.269388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0463fa5f-4621-4e6a-893e-5df125d67cb7-iptables-alerter-script\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.276849 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.276833 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:45.276897 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.276854 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:45.276897 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.276864 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:45.276989 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.276936 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:46:45.776905447 +0000 UTC m=+2.015462659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:45.277039 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.276996 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 15:46:45.279789 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.279774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjztg\" (UniqueName: \"kubernetes.io/projected/420ce120-0f79-4acc-89cd-a53bdf44543d-kube-api-access-bjztg\") pod \"aws-ebs-csi-driver-node-cwxjw\" (UID: \"420ce120-0f79-4acc-89cd-a53bdf44543d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.284910 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.284890 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4dj\" (UniqueName: \"kubernetes.io/projected/0463fa5f-4621-4e6a-893e-5df125d67cb7-kube-api-access-zc4dj\") pod \"iptables-alerter-26n6r\" (UID: \"0463fa5f-4621-4e6a-893e-5df125d67cb7\") " pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.323707 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.323681 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fba2a84e38e9d2ca29be30525318bd.slice/crio-91a91dd5a3c95f4751a5aae35be66d8ab03e39ee4af26085ee490dcbeda66068 WatchSource:0}: Error finding container 91a91dd5a3c95f4751a5aae35be66d8ab03e39ee4af26085ee490dcbeda66068: Status 404 returned error can't find the container with id 91a91dd5a3c95f4751a5aae35be66d8ab03e39ee4af26085ee490dcbeda66068 Jul 29 15:46:45.329306 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.329267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"06fba2a84e38e9d2ca29be30525318bd","Type":"ContainerStarted","Data":"91a91dd5a3c95f4751a5aae35be66d8ab03e39ee4af26085ee490dcbeda66068"} Jul 29 15:46:45.330121 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.330102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" event={"ID":"c7c1d29852edb376853ce02192f5c250","Type":"ContainerStarted","Data":"b3a548c8eeba8399aaee20e282f3f912c914b166f51b514d57cb7b8d3c6fee95"} Jul 29 15:46:45.369689 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-multus-certs\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.369689 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57zs\" (UniqueName: \"kubernetes.io/projected/98be5804-426c-4fa2-883c-0e9e0208cbd4-kube-api-access-b57zs\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-lib-modules\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-tuned\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-multus-certs\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-netns\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-netns\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-kubelet\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-host\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.369862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bb48\" (UniqueName: \"kubernetes.io/projected/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-kube-api-access-4bb48\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-systemd-units\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-etc-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-lib-modules\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-kubelet\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-log-socket\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-etc-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-systemd-units\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-log-socket\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.369970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-host\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-conf\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-slash\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-systemd\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-slash\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-run\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-conf\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.370231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-sys\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-k8s-cni-cncf-io\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-tmp\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-sys\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-systemd\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4961fd-5883-4b04-8eb0-1c950898d3ab-tmp-dir\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-run-k8s-cni-cncf-io\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-run\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-ovn\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-host\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-cnibin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-multus\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-ovn\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4961fd-5883-4b04-8eb0-1c950898d3ab-hosts-file\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-daemon-config\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-system-cni-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-cnibin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4961fd-5883-4b04-8eb0-1c950898d3ab-tmp-dir\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-host\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af4961fd-5883-4b04-8eb0-1c950898d3ab-hosts-file\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.370929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-systemd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-node-log\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371316 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-node-log\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-systemd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-system-cni-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-cni-binary-copy\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-socket-dir-parent\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371606 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-multus\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-bin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-daemon-config\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.371901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-var-lib-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkc4\" (UniqueName: \"kubernetes.io/projected/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-kube-api-access-jxkc4\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-conf-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrct\" (UniqueName: \"kubernetes.io/projected/40634acf-43ac-4123-81b1-9136df1c4d6b-kube-api-access-6qrct\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.372739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.371902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-conf-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.373345 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.373326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-tuned\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.373419 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.373386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-tmp\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.373488 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.373472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-host-var-lib-cni-bin\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.373584 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.373567 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:45.373701 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.373689 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:46:45.873663394 +0000 UTC m=+2.112220584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:45.374000 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.373951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-multus-socket-dir-parent\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.374161 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-var-lib-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374228 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374278 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63d1846d-8f08-4e0e-bd53-4670efb97c85-agent-certs\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.374278 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.374364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-var-lib-kubelet\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.374364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-env-overrides\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-kubelet-config\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.374498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-hostroot\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.374498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc6l\" (UniqueName: \"kubernetes.io/projected/648d11ed-cc61-426a-913f-cde108d6b63c-kube-api-access-cpc6l\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.374498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-kubelet\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-netd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovn-node-metrics-cert\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-run-openvswitch\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kglw\" (UniqueName: \"kubernetes.io/projected/839cb74e-cea4-4bfb-99fb-010af6495d0b-kube-api-access-8kglw\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-kubelet\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-serviceca\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-bin\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374628 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-kubelet-config\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.374667 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-script-lib\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374669 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-hostroot\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-dbus\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-modprobe-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40634acf-43ac-4123-81b1-9136df1c4d6b-cni-binary-copy\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-netd\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-var-lib-kubelet\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-cni-bin\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysctl-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-modprobe-d\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/901c05b2-86a2-4a19-9e37-99b5f4b79c26-dbus\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.375083 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.374941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysconfig\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-netns\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-etc-kubernetes\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqbp\" (UniqueName: \"kubernetes.io/projected/af4961fd-5883-4b04-8eb0-1c950898d3ab-kube-api-access-vlqbp\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-serviceca\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-cnibin\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-os-release\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-config\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-os-release\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63d1846d-8f08-4e0e-bd53-4670efb97c85-konnectivity-ca\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375429 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98be5804-426c-4fa2-883c-0e9e0208cbd4-host-run-netns\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.375496 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-kubernetes\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-os-release\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.375784 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-system-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-sysconfig\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.375764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-etc-kubernetes\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40634acf-43ac-4123-81b1-9136df1c4d6b-system-cni-dir\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.376070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376064 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-env-overrides\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.376137 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:45.876107637 +0000 UTC m=+2.114664839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-cnibin\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-etc-kubernetes\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376284 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/648d11ed-cc61-426a-913f-cde108d6b63c-os-release\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-config\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.376451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovnkube-script-lib\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.376811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/648d11ed-cc61-426a-913f-cde108d6b63c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.376811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63d1846d-8f08-4e0e-bd53-4670efb97c85-konnectivity-ca\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.376880 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.376855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63d1846d-8f08-4e0e-bd53-4670efb97c85-agent-certs\") pod \"konnectivity-agent-kzhbk\" (UID: \"63d1846d-8f08-4e0e-bd53-4670efb97c85\") " pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.377629 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.377611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98be5804-426c-4fa2-883c-0e9e0208cbd4-ovn-node-metrics-cert\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.379149 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.379125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bb48\" (UniqueName: \"kubernetes.io/projected/a1f2f29e-edf9-4cd8-a20c-11f5902c28e2-kube-api-access-4bb48\") pod \"tuned-9fs2g\" (UID: \"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2\") " pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.379717 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.379697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkc4\" (UniqueName: \"kubernetes.io/projected/2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4-kube-api-access-jxkc4\") pod \"node-ca-5ptlz\" (UID: \"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4\") " pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.380124 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.380108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57zs\" (UniqueName: \"kubernetes.io/projected/98be5804-426c-4fa2-883c-0e9e0208cbd4-kube-api-access-b57zs\") pod \"ovnkube-node-r4hgq\" (UID: \"98be5804-426c-4fa2-883c-0e9e0208cbd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.386522 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.386478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kglw\" (UniqueName: \"kubernetes.io/projected/839cb74e-cea4-4bfb-99fb-010af6495d0b-kube-api-access-8kglw\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.390922 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.390902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrct\" (UniqueName: \"kubernetes.io/projected/40634acf-43ac-4123-81b1-9136df1c4d6b-kube-api-access-6qrct\") pod \"multus-vx7tl\" (UID: \"40634acf-43ac-4123-81b1-9136df1c4d6b\") " pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.391135 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.391119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqbp\" (UniqueName: \"kubernetes.io/projected/af4961fd-5883-4b04-8eb0-1c950898d3ab-kube-api-access-vlqbp\") pod \"node-resolver-994kz\" (UID: \"af4961fd-5883-4b04-8eb0-1c950898d3ab\") " pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.391187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.391152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc6l\" (UniqueName: \"kubernetes.io/projected/648d11ed-cc61-426a-913f-cde108d6b63c-kube-api-access-cpc6l\") pod \"multus-additional-cni-plugins-s5cv4\" (UID: \"648d11ed-cc61-426a-913f-cde108d6b63c\") " pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.489017 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.488988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26n6r" Jul 29 15:46:45.497285 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.497264 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0463fa5f_4621_4e6a_893e_5df125d67cb7.slice/crio-4e88e023be4ae6ed7b3d0ff15e8ad2022c86075631f88e647b46e2747b33b6fd WatchSource:0}: Error finding container 4e88e023be4ae6ed7b3d0ff15e8ad2022c86075631f88e647b46e2747b33b6fd: Status 404 returned error can't find the container with id 4e88e023be4ae6ed7b3d0ff15e8ad2022c86075631f88e647b46e2747b33b6fd Jul 29 15:46:45.509911 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.509892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" Jul 29 15:46:45.517090 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.517068 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420ce120_0f79_4acc_89cd_a53bdf44543d.slice/crio-263e2d5fdc99fa272449c16fefb56061136ac5c652226e9e9f4788db2aecb1e5 WatchSource:0}: Error finding container 263e2d5fdc99fa272449c16fefb56061136ac5c652226e9e9f4788db2aecb1e5: Status 404 returned error can't find the container with id 263e2d5fdc99fa272449c16fefb56061136ac5c652226e9e9f4788db2aecb1e5 Jul 29 15:46:45.530559 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.530535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" Jul 29 15:46:45.536608 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.536590 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f2f29e_edf9_4cd8_a20c_11f5902c28e2.slice/crio-27d9c38fa7ac931af89311b75e6ba5f65a3b3f5c38b238b2e8b98e247a50a666 WatchSource:0}: Error finding container 27d9c38fa7ac931af89311b75e6ba5f65a3b3f5c38b238b2e8b98e247a50a666: Status 404 returned error can't find the container with id 27d9c38fa7ac931af89311b75e6ba5f65a3b3f5c38b238b2e8b98e247a50a666 Jul 29 15:46:45.538216 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.538191 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ptlz" Jul 29 15:46:45.545659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.545640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vx7tl" Jul 29 15:46:45.546345 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.546328 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb8fbe2_7a23_421c_ab82_7de86ba3d6b4.slice/crio-859d99d3c4527bb9976677d41ede77dfadf68e85e703b2323ab2c615c31a5199 WatchSource:0}: Error finding container 859d99d3c4527bb9976677d41ede77dfadf68e85e703b2323ab2c615c31a5199: Status 404 returned error can't find the container with id 859d99d3c4527bb9976677d41ede77dfadf68e85e703b2323ab2c615c31a5199 Jul 29 15:46:45.550119 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.550101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-994kz" Jul 29 15:46:45.551926 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.551896 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40634acf_43ac_4123_81b1_9136df1c4d6b.slice/crio-a18ee46bce34d02af48e707db9a9f10ca8d7c17fa658268f5c46f8940eee87c8 WatchSource:0}: Error finding container a18ee46bce34d02af48e707db9a9f10ca8d7c17fa658268f5c46f8940eee87c8: Status 404 returned error can't find the container with id a18ee46bce34d02af48e707db9a9f10ca8d7c17fa658268f5c46f8940eee87c8 Jul 29 15:46:45.556253 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.556235 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:46:45.557569 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.557550 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4961fd_5883_4b04_8eb0_1c950898d3ab.slice/crio-d8c1f5c31347bae2f92f71b251898d8d69eadfb98b0e116b2bd7ab8f8cc42d49 WatchSource:0}: Error finding container d8c1f5c31347bae2f92f71b251898d8d69eadfb98b0e116b2bd7ab8f8cc42d49: Status 404 returned error can't find the container with id d8c1f5c31347bae2f92f71b251898d8d69eadfb98b0e116b2bd7ab8f8cc42d49 Jul 29 15:46:45.561463 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.561447 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:46:45.563189 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.563170 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d1846d_8f08_4e0e_bd53_4670efb97c85.slice/crio-2a1e7ec4f01d22f3a824a2520b53d3fc556792ed83c2fcf1402616c127b46e90 WatchSource:0}: Error finding container 2a1e7ec4f01d22f3a824a2520b53d3fc556792ed83c2fcf1402616c127b46e90: Status 404 returned error can't find the container with id 2a1e7ec4f01d22f3a824a2520b53d3fc556792ed83c2fcf1402616c127b46e90 Jul 29 15:46:45.565501 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.565478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" Jul 29 15:46:45.568214 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.568194 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98be5804_426c_4fa2_883c_0e9e0208cbd4.slice/crio-34c6e05b6d52e686bf3bd88ad77178b3a7876529f476ffc26892126cda013554 WatchSource:0}: Error finding container 34c6e05b6d52e686bf3bd88ad77178b3a7876529f476ffc26892126cda013554: Status 404 returned error can't find the container with id 34c6e05b6d52e686bf3bd88ad77178b3a7876529f476ffc26892126cda013554 Jul 29 15:46:45.572928 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:46:45.572901 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648d11ed_cc61_426a_913f_cde108d6b63c.slice/crio-6b8141063b2d2a9468b81e1b7f22f0f9d17b18b2a0a5403fe82225f9c15470ae WatchSource:0}: Error finding container 6b8141063b2d2a9468b81e1b7f22f0f9d17b18b2a0a5403fe82225f9c15470ae: Status 404 returned error can't find the container with id 6b8141063b2d2a9468b81e1b7f22f0f9d17b18b2a0a5403fe82225f9c15470ae Jul 29 15:46:45.780241 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.780160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:45.780391 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.780315 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:45.780391 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.780337 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:45.780391 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.780350 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:45.780523 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.780407 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.78038838 +0000 UTC m=+3.018945574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:45.880681 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.880637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:45.881027 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:45.880708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:45.881027 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.880866 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:45.881027 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.880941 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.880910169 +0000 UTC m=+3.119467381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:45.881336 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.881315 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:45.881427 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:45.881383 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:46:46.881365095 +0000 UTC m=+3.119922287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:46.224816 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.224702 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 15:41:45 +0000 UTC" deadline="2028-03-11 22:25:21.01450145 +0000 UTC" Jul 29 15:46:46.224816 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.224780 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14190h38m34.789739978s" Jul 29 15:46:46.342660 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.342622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kzhbk" event={"ID":"63d1846d-8f08-4e0e-bd53-4670efb97c85","Type":"ContainerStarted","Data":"2a1e7ec4f01d22f3a824a2520b53d3fc556792ed83c2fcf1402616c127b46e90"} Jul 29 15:46:46.347692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.347657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-994kz" event={"ID":"af4961fd-5883-4b04-8eb0-1c950898d3ab","Type":"ContainerStarted","Data":"d8c1f5c31347bae2f92f71b251898d8d69eadfb98b0e116b2bd7ab8f8cc42d49"} Jul 29 15:46:46.352087 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.352056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vx7tl" event={"ID":"40634acf-43ac-4123-81b1-9136df1c4d6b","Type":"ContainerStarted","Data":"a18ee46bce34d02af48e707db9a9f10ca8d7c17fa658268f5c46f8940eee87c8"} Jul 29 15:46:46.361564 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.361538 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ptlz" event={"ID":"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4","Type":"ContainerStarted","Data":"859d99d3c4527bb9976677d41ede77dfadf68e85e703b2323ab2c615c31a5199"} Jul 29 15:46:46.365262 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.365194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" event={"ID":"420ce120-0f79-4acc-89cd-a53bdf44543d","Type":"ContainerStarted","Data":"263e2d5fdc99fa272449c16fefb56061136ac5c652226e9e9f4788db2aecb1e5"} Jul 29 15:46:46.367572 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.367499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26n6r" event={"ID":"0463fa5f-4621-4e6a-893e-5df125d67cb7","Type":"ContainerStarted","Data":"4e88e023be4ae6ed7b3d0ff15e8ad2022c86075631f88e647b46e2747b33b6fd"} Jul 29 15:46:46.369465 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.369385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerStarted","Data":"6b8141063b2d2a9468b81e1b7f22f0f9d17b18b2a0a5403fe82225f9c15470ae"} Jul 29 15:46:46.380760 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.380659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" event={"ID":"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2","Type":"ContainerStarted","Data":"27d9c38fa7ac931af89311b75e6ba5f65a3b3f5c38b238b2e8b98e247a50a666"} Jul 29 15:46:46.383503 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.383394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"34c6e05b6d52e686bf3bd88ad77178b3a7876529f476ffc26892126cda013554"} Jul 29 15:46:46.434960 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.434740 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:46.610376 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.610296 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:46.789287 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.789245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:46.789498 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.789481 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:46.789563 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.789510 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:46.789563 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.789523 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:46.789683 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.789596 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:46:48.78957592 +0000 UTC m=+5.028133132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.890405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:46.890485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.890609 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.890673 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:46:48.890653049 +0000 UTC m=+5.129210240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.891075 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:46.891236 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:46.891133 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:48.891118216 +0000 UTC m=+5.129675410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:47.200497 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.197358 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 15:46:47.225961 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.225903 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 15:41:45 +0000 UTC" deadline="2028-01-20 04:19:47.495899227 +0000 UTC" Jul 29 15:46:47.225961 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.225959 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12948h33m0.269944062s" Jul 29 15:46:47.327592 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.327561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:47.327767 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:47.327708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:47.327828 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.327801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:47.328008 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:47.327872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:47.328008 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:47.327926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:47.328136 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:47.328005 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:48.808231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:48.808149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:48.808684 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.808311 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:48.808684 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.808341 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:48.808684 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.808355 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:48.808684 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.808415 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:46:52.808393602 +0000 UTC m=+9.046950807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:48.909164 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:48.909128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:48.909329 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:48.909237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:48.909436 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.909418 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:48.909503 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.909468 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:46:52.909455438 +0000 UTC m=+9.148012629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:48.909764 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.909750 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:48.909819 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:48.909789 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:46:52.909779265 +0000 UTC m=+9.148336452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:49.327267 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:49.327162 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:49.327267 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:49.327212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:49.327267 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:49.327212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:49.327554 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:49.327289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:49.327554 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:49.327421 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:49.327554 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:49.327499 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:51.327341 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:51.327304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:51.327807 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:51.327344 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:51.327807 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:51.327461 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:51.327807 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:51.327545 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:51.327807 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:51.327619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:51.327807 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:51.327692 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:52.843929 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:52.843891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:52.844409 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.844061 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:46:52.844409 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.844086 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:46:52.844409 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.844099 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:52.844409 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.844169 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:47:00.844150245 +0000 UTC m=+17.082707437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:46:52.944466 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:52.944383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:52.944466 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:52.944454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:52.944698 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.944599 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:52.944748 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.944705 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:00.944685674 +0000 UTC m=+17.183242877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:46:52.945009 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.944970 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:52.945144 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:52.945049 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:47:00.945030855 +0000 UTC m=+17.183588047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:46:53.327552 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:53.327470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:53.327716 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:53.327619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:53.328141 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:53.328008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:53.328141 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:53.328103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:53.328322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:53.328148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:53.328322 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:53.328253 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:55.328016 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:55.327705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:55.328016 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:55.327752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:55.328016 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:55.327752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:55.328016 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:55.327820 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:55.328016 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:55.327941 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:55.328621 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:55.328071 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:57.327859 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:57.327824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:57.328271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:57.327868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:57.328271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:57.327823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:57.328271 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:57.327952 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:57.328271 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:57.328017 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:57.328271 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:57.328086 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:46:59.327163 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:59.327123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:46:59.327572 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:59.327133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:46:59.327572 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:46:59.327253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:46:59.327572 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:59.327254 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:46:59.327572 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:59.327349 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:46:59.327572 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:46:59.327440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:00.905898 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:00.905866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:00.906317 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:00.906016 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:47:00.906317 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:00.906035 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:47:00.906317 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:00.906048 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:00.906317 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:00.906100 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:47:16.906084868 +0000 UTC m=+33.144642057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:01.006395 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:01.006360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:01.006564 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:01.006439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:01.006564 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.006525 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:01.006564 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.006552 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:01.006728 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.006605 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.00658347 +0000 UTC m=+33.245140659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:01.006728 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.006626 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:17.006615771 +0000 UTC m=+33.245172964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:01.327463 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:01.327434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:01.327620 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:01.327566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:01.327620 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:01.327592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:01.327732 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.327664 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:01.327732 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.327567 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:01.327801 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:01.327766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:03.327431 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:03.327389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:03.327849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:03.327442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:03.327849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:03.327517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:03.327849 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:03.327539 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:03.327849 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:03.327596 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:03.327849 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:03.327665 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:05.327744 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.327467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:05.328465 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:05.327895 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:05.328465 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.327495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:05.328465 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.327522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:05.328465 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:05.328034 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:05.328465 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:05.328047 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:05.422507 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.422474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" event={"ID":"c7c1d29852edb376853ce02192f5c250","Type":"ContainerStarted","Data":"032f83e1c97605e1e03f4d43246d779ba9ab2cdde76dab06c7e72f39431e1879"} Jul 29 15:47:05.424838 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.424820 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:47:05.425187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425163 2574 generic.go:356] "Generic (PLEG): container finished" podID="98be5804-426c-4fa2-883c-0e9e0208cbd4" containerID="1e3541481f190ff6dec4e581c227b052113292f9ced70a8b7a98b480fa240337" exitCode=1 Jul 29 15:47:05.425274 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"b8aa581a9de9cdbfe44139b7c2f978d53604425a3940baab6b999b9cc59f5aa5"} Jul 29 15:47:05.425326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"0f78667c64c2ebd79f54f17594cb59549cb2e1f1800e4fd9368ed94d3de0fce7"} Jul 29 15:47:05.425368 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"52d788978aeca3ea62bd32296dfb738ddbe353cbe42ee53acf8fc7908ddc12a3"} Jul 29 15:47:05.425368 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerDied","Data":"1e3541481f190ff6dec4e581c227b052113292f9ced70a8b7a98b480fa240337"} Jul 29 15:47:05.425440 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.425398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"bb1022f9d7ca5d1a75c760ec6055a77bf20414fad15dd2fb43614e7ecda42fca"} Jul 29 15:47:05.426551 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.426532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kzhbk" event={"ID":"63d1846d-8f08-4e0e-bd53-4670efb97c85","Type":"ContainerStarted","Data":"fcdf45bad288bc7ab9caab6be1e6392eeef3377a9f0a6151142176959dcca9ec"} Jul 29 15:47:05.427791 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.427770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-994kz" event={"ID":"af4961fd-5883-4b04-8eb0-1c950898d3ab","Type":"ContainerStarted","Data":"ced8241c5357d81a81738db0ff3951ea8f1a5b010b5b63b95244a570ff80af94"} Jul 29 15:47:05.429093 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.429075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vx7tl" event={"ID":"40634acf-43ac-4123-81b1-9136df1c4d6b","Type":"ContainerStarted","Data":"6f402a213d570fb81ef2fcf6e8bc948626a9b0ee31e5e089366ce481545e400e"} Jul 29 15:47:05.430393 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.430368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ptlz" event={"ID":"2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4","Type":"ContainerStarted","Data":"e572d87915ab52fe5ca1e1d69a1a58384bbba2284996abf119595cda1e5d5700"} Jul 29 15:47:05.431631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.431610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" event={"ID":"420ce120-0f79-4acc-89cd-a53bdf44543d","Type":"ContainerStarted","Data":"33f29d3dc5ae630f0813949102d1c6268d1a3a9ab7d655b6f0ad7205845b19b9"} Jul 29 15:47:05.432937 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.432912 2574 generic.go:356] "Generic (PLEG): container finished" podID="06fba2a84e38e9d2ca29be30525318bd" containerID="7bad3808424a0165666fd98ed05a42b25ec061e9a56fa926724eeef1363b31f1" exitCode=0 Jul 29 15:47:05.433031 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.432992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"06fba2a84e38e9d2ca29be30525318bd","Type":"ContainerDied","Data":"7bad3808424a0165666fd98ed05a42b25ec061e9a56fa926724eeef1363b31f1"} Jul 29 15:47:05.434339 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.434320 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="681283564f451cf162e3da18ccff8b37a1c6fdd82ee3bf3e7739771fdba783de" exitCode=0 Jul 29 15:47:05.434411 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.434383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"681283564f451cf162e3da18ccff8b37a1c6fdd82ee3bf3e7739771fdba783de"} Jul 29 15:47:05.435757 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.435741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" event={"ID":"a1f2f29e-edf9-4cd8-a20c-11f5902c28e2","Type":"ContainerStarted","Data":"cb868b7b1a5715ee0b7792a14fad40eb854b6a3db3cd94b7f25cf8b6caf64359"} Jul 29 15:47:05.437106 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.437024 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" podStartSLOduration=20.437012924 podStartE2EDuration="20.437012924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.436466798 +0000 UTC m=+21.675023998" watchObservedRunningTime="2026-07-29 15:47:05.437012924 +0000 UTC m=+21.675570133" Jul 29 15:47:05.454832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.452896 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-994kz" podStartSLOduration=2.448225351 podStartE2EDuration="21.452878351s" totalImagesPullingTime="19.004653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.452550897 +0000 UTC m=+21.691108097" watchObservedRunningTime="2026-07-29 15:47:05.452878351 +0000 UTC m=+21.691435563" Jul 29 15:47:05.504633 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.504454 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9fs2g" podStartSLOduration=2.4803298910000002 podStartE2EDuration="21.504438673s" totalImagesPullingTime="19.024108782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.503971723 +0000 UTC m=+21.742528937" watchObservedRunningTime="2026-07-29 15:47:05.504438673 +0000 UTC m=+21.742995900" Jul 29 15:47:05.521962 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.521904 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5ptlz" podStartSLOduration=2.554353567 podStartE2EDuration="21.521884652s" totalImagesPullingTime="18.967531085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.521390593 +0000 UTC m=+21.759947804" watchObservedRunningTime="2026-07-29 15:47:05.521884652 +0000 UTC m=+21.760441864" Jul 29 15:47:05.545860 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.545809 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vx7tl" podStartSLOduration=2.013381833 podStartE2EDuration="21.545788359s" totalImagesPullingTime="19.532406526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.54522095 +0000 UTC m=+21.783778163" watchObservedRunningTime="2026-07-29 15:47:05.545788359 +0000 UTC m=+21.784345569" Jul 29 15:47:05.562881 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.562830 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kzhbk" podStartSLOduration=6.7052175290000005 podStartE2EDuration="21.562813913s" totalImagesPullingTime="14.857596384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:05.562343454 +0000 UTC m=+21.800900666" watchObservedRunningTime="2026-07-29 15:47:05.562813913 +0000 UTC m=+21.801371123" Jul 29 15:47:05.999522 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:05.999494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:06.065790 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.065754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:06.439840 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.439750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26n6r" event={"ID":"0463fa5f-4621-4e6a-893e-5df125d67cb7","Type":"ContainerStarted","Data":"039bafa74801260ffe00a906876f32d98807a39831c6d96cf1751f7924f518bb"} Jul 29 15:47:06.441821 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.441790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"06fba2a84e38e9d2ca29be30525318bd","Type":"ContainerStarted","Data":"6751723b4b563824688fc90a2dc7e543b77a6ad806aaf3425b2ec94f1bea0d87"} Jul 29 15:47:06.444825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.444799 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:47:06.445277 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.445253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"3df850dd4796aa74f0013f7f3df626deeb0b12f63b965ee0abcb63b2285d053f"} Jul 29 15:47:06.456097 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.456053 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-26n6r" podStartSLOduration=3.4395665060000002 podStartE2EDuration="22.456037094s" totalImagesPullingTime="19.016470588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:06.455951685 +0000 UTC m=+22.694508896" watchObservedRunningTime="2026-07-29 15:47:06.456037094 +0000 UTC m=+22.694594305" Jul 29 15:47:06.470781 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.470736 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" podStartSLOduration=21.470722717 podStartE2EDuration="21.470722717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:06.470607827 +0000 UTC m=+22.709165037" watchObservedRunningTime="2026-07-29 15:47:06.470722717 +0000 UTC m=+22.709279926" Jul 29 15:47:06.616663 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.616624 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 15:47:06.768758 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.768724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:06.832808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:06.832763 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:07.234402 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.234241 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T15:47:06.616644414Z","UUID":"bd07d595-98a2-4543-9e37-14439f8408b1","Handler":null,"Name":"","Endpoint":""} Jul 29 15:47:07.235787 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.235769 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 15:47:07.235916 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.235806 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 15:47:07.326959 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.326929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:07.327213 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:07.327062 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:07.327213 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.327080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:07.327213 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:07.327175 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:07.327213 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.327073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:07.327421 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:07.327294 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:07.449550 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:07.449512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" event={"ID":"420ce120-0f79-4acc-89cd-a53bdf44543d","Type":"ContainerStarted","Data":"dd3c26d7374f07f0dc52a4e7b0d5f32550eeb6c15378675a53ac1676188d59f1"} Jul 29 15:47:08.266345 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.266316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:08.335489 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.335448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:08.440162 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.440129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:08.454792 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.454765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:47:08.455216 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.455142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"4e2d170d8cb29dee347e0a2e1b78bc67a638b0eae2fe84091b85b34822fb133b"} Jul 29 15:47:08.539569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:08.539483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:09.327765 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:09.327734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:09.327946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:09.327787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:09.327946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:09.327914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:09.327946 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:09.327927 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:09.328083 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:09.328050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:09.328165 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:09.328144 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:10.063862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.063680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:10.131913 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.131834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:10.160824 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.160799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:47:10.161383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.161367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:47:10.462443 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.462416 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:47:10.462818 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.462785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"37174abd70a7a495b2e6efcf5010e074c95a067c70b265edcea56319ae721309"} Jul 29 15:47:10.463080 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.463056 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:10.463173 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.463129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:10.463266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.463249 2574 scope.go:117] "RemoveContainer" containerID="1e3541481f190ff6dec4e581c227b052113292f9ced70a8b7a98b480fa240337" Jul 29 15:47:10.465116 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.465082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" event={"ID":"420ce120-0f79-4acc-89cd-a53bdf44543d","Type":"ContainerStarted","Data":"3844c991f792f8b326191826bd22b059f09d3023f5ab74381300ef7c69274444"} Jul 29 15:47:10.466872 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.466850 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="3ced6ba8d51457429e6911a20be952d20cc902ab37985f731ef4b71de2927871" exitCode=0 Jul 29 15:47:10.466958 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.466923 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"3ced6ba8d51457429e6911a20be952d20cc902ab37985f731ef4b71de2927871"} Jul 29 15:47:10.467146 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.467132 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:47:10.467587 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.467544 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kzhbk" Jul 29 15:47:10.479076 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.479059 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:10.560225 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.560179 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwxjw" podStartSLOduration=3.351953834 podStartE2EDuration="26.560163226s" totalImagesPullingTime="23.208209392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:10.559822607 +0000 UTC m=+26.798379818" watchObservedRunningTime="2026-07-29 15:47:10.560163226 +0000 UTC m=+26.798720435" Jul 29 15:47:10.977310 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:10.977270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:11.047645 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.047602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:11.327461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.327280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:11.327923 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.327319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:11.327923 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.327510 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:11.327923 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.327319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:11.327923 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.327570 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:11.327923 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.327637 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:11.473964 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.473941 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:47:11.474406 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.474375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" event={"ID":"98be5804-426c-4fa2-883c-0e9e0208cbd4","Type":"ContainerStarted","Data":"abeb4e077fcf2c0439231880e21273aab6bfd9e659e26336782f5e74d8795f41"} Jul 29 15:47:11.474631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.474606 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:11.491046 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.491018 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:11.507857 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.507806 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" podStartSLOduration=8.514972691 podStartE2EDuration="27.507787109s" totalImagesPullingTime="18.992814418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:11.50616897 +0000 UTC m=+27.744726205" watchObservedRunningTime="2026-07-29 15:47:11.507787109 +0000 UTC m=+27.746344320" Jul 29 15:47:11.980391 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.980312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9k2g"] Jul 29 15:47:11.980567 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.980421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:11.980567 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.980520 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:11.983639 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.983608 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7224n"] Jul 29 15:47:11.983763 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.983705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:11.983821 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.983796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:11.984281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.984258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmxs9"] Jul 29 15:47:11.984379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:11.984335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:11.984447 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:11.984414 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:12.478413 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:12.478382 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="6181d1a5f8c6d91a0f476e877b28f6510fb9eb19bf7d2a75ea89e92f96f5d2ef" exitCode=0 Jul 29 15:47:12.478860 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:12.478466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"6181d1a5f8c6d91a0f476e877b28f6510fb9eb19bf7d2a75ea89e92f96f5d2ef"} Jul 29 15:47:13.326928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:13.326899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:13.327079 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:13.327013 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:14.329811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:14.329785 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:14.330246 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:14.329910 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:14.330246 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:14.329951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:14.330246 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:14.330044 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:14.484491 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:14.484462 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="1c84aa0c8ff27981018a4e19516189241a0849616b67812dc312fa4c08a98e8e" exitCode=0 Jul 29 15:47:14.484635 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:14.484499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"1c84aa0c8ff27981018a4e19516189241a0849616b67812dc312fa4c08a98e8e"} Jul 29 15:47:15.327266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:15.327083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:15.327453 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:15.327400 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:15.472995 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:15.472944 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:15.549943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:15.549887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:16.327191 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:16.327167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:16.327369 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:16.327167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:16.327369 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.327313 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmxs9" podUID="839cb74e-cea4-4bfb-99fb-010af6495d0b" Jul 29 15:47:16.327491 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.327440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9k2g" podUID="901c05b2-86a2-4a19-9e37-99b5f4b79c26" Jul 29 15:47:16.398134 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:16.398101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:16.931219 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:16.931178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:16.931621 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.931374 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 15:47:16.931621 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.931401 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 15:47:16.931621 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.931416 2574 projected.go:196] Error preparing data for projected volume kube-api-access-rj6k4 for pod openshift-network-diagnostics/network-check-target-7224n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:16.931621 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:16.931471 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4 podName:f48a0452-0f23-4911-aa5f-dddc65ea5d5e nodeName:}" failed. No retries permitted until 2026-07-29 15:47:48.931453914 +0000 UTC m=+65.170011116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6k4" (UniqueName: "kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4") pod "network-check-target-7224n" (UID: "f48a0452-0f23-4911-aa5f-dddc65ea5d5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 15:47:17.032396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.032347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:17.032560 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.032416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:17.032560 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.032535 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:17.032670 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.032595 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs podName:839cb74e-cea4-4bfb-99fb-010af6495d0b nodeName:}" failed. No retries permitted until 2026-07-29 15:47:49.03257976 +0000 UTC m=+65.271136949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs") pod "network-metrics-daemon-jmxs9" (UID: "839cb74e-cea4-4bfb-99fb-010af6495d0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 15:47:17.032670 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.032536 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:17.032766 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.032688 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret podName:901c05b2-86a2-4a19-9e37-99b5f4b79c26 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:49.032666863 +0000 UTC m=+65.271224068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret") pod "global-pull-secret-syncer-w9k2g" (UID: "901c05b2-86a2-4a19-9e37-99b5f4b79c26") : object "kube-system"/"original-pull-secret" not registered Jul 29 15:47:17.320436 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.320400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:17.327485 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.327462 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:17.327632 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.327576 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7224n" podUID="f48a0452-0f23-4911-aa5f-dddc65ea5d5e" Jul 29 15:47:17.431320 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.431286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:17.571278 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.571217 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeReady" Jul 29 15:47:17.571393 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.571356 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 15:47:17.649675 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.649646 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rwjjb"] Jul 29 15:47:17.652774 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.652215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.653184 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.653159 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vwgt7"] Jul 29 15:47:17.654791 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.654769 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.656237 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.656217 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx57w\"" Jul 29 15:47:17.657574 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.657555 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 15:47:17.658081 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.658060 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 15:47:17.658189 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.658134 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 15:47:17.658618 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.658601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 15:47:17.658713 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.658649 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h46bm\"" Jul 29 15:47:17.659133 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.659106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 15:47:17.672701 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.672674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vwgt7"] Jul 29 15:47:17.673512 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.673490 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwjjb"] Jul 29 15:47:17.737117 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/750ea16e-e156-4180-b5f8-067086f46f82-cert\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.737275 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.737275 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-config-volume\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.737275 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-kube-api-access-vvnxk\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.737275 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-tmp-dir\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.737480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.737286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8tr\" (UniqueName: \"kubernetes.io/projected/750ea16e-e156-4180-b5f8-067086f46f82-kube-api-access-dl8tr\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.821677 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.821591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:17.839087 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.838839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/750ea16e-e156-4180-b5f8-067086f46f82-cert\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.839244 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.839244 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-config-volume\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.839244 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-kube-api-access-vvnxk\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.839383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-tmp-dir\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.839383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8tr\" (UniqueName: \"kubernetes.io/projected/750ea16e-e156-4180-b5f8-067086f46f82-kube-api-access-dl8tr\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.840079 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.839764 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:17.840079 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:17.839840 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls podName:6cb7b2da-358a-4963-8d3c-0d31f5db64bb nodeName:}" failed. No retries permitted until 2026-07-29 15:47:18.339819703 +0000 UTC m=+34.578376895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls") pod "dns-default-rwjjb" (UID: "6cb7b2da-358a-4963-8d3c-0d31f5db64bb") : secret "dns-default-metrics-tls" not found Jul 29 15:47:17.840079 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.839898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-tmp-dir\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.840298 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.840180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-config-volume\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.844626 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.844606 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/750ea16e-e156-4180-b5f8-067086f46f82-cert\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.850397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.850371 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-kube-api-access-vvnxk\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:17.851104 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.851082 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8tr\" (UniqueName: \"kubernetes.io/projected/750ea16e-e156-4180-b5f8-067086f46f82-kube-api-access-dl8tr\") pod \"ingress-canary-vwgt7\" (UID: \"750ea16e-e156-4180-b5f8-067086f46f82\") " pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:17.918233 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.918201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:17.972455 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:17.972420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vwgt7" Jul 29 15:47:18.327298 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.327263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:18.327477 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.327445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:18.330231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.330206 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 15:47:18.331288 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.331268 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 15:47:18.331396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.331271 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28zhr\"" Jul 29 15:47:18.344070 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.344046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:18.344216 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:18.344192 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:18.344324 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:18.344272 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls podName:6cb7b2da-358a-4963-8d3c-0d31f5db64bb nodeName:}" failed. No retries permitted until 2026-07-29 15:47:19.34425157 +0000 UTC m=+35.582808760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls") pod "dns-default-rwjjb" (UID: "6cb7b2da-358a-4963-8d3c-0d31f5db64bb") : secret "dns-default-metrics-tls" not found Jul 29 15:47:18.634387 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.634236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:18.662892 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.662793 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vwgt7"] Jul 29 15:47:18.665093 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:18.665063 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750ea16e_e156_4180_b5f8_067086f46f82.slice/crio-e11a52b364c2732ee9ed68265163eb40e4e73f9450c54da484426fbc89c9ddd3 WatchSource:0}: Error finding container e11a52b364c2732ee9ed68265163eb40e4e73f9450c54da484426fbc89c9ddd3: Status 404 returned error can't find the container with id e11a52b364c2732ee9ed68265163eb40e4e73f9450c54da484426fbc89c9ddd3 Jul 29 15:47:18.778770 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.778741 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v4h77"] Jul 29 15:47:18.780648 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.780634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:18.785171 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.785152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 15:47:18.785383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.785352 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 15:47:18.785478 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.785380 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 15:47:18.785478 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.785397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sv279\"" Jul 29 15:47:18.785583 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.785482 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 15:47:18.802157 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.802138 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4h77"] Jul 29 15:47:18.948212 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.948124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:18.948337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.948226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d5aebed-f552-4edf-ac20-82310dd6cf44-data-volume\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:18.948337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.948248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjz7k\" (UniqueName: \"kubernetes.io/projected/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-api-access-pjz7k\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:18.948337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.948276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d5aebed-f552-4edf-ac20-82310dd6cf44-crio-socket\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:18.948337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:18.948303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.049362 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d5aebed-f552-4edf-ac20-82310dd6cf44-data-volume\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz7k\" (UniqueName: \"kubernetes.io/projected/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-api-access-pjz7k\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d5aebed-f552-4edf-ac20-82310dd6cf44-crio-socket\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.049485 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.049519 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.049562 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:19.549540699 +0000 UTC m=+35.788097906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.049582 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:19.549572451 +0000 UTC m=+35.788129642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : configmap "kube-rbac-proxy" not found Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d5aebed-f552-4edf-ac20-82310dd6cf44-crio-socket\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.050042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.049822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d5aebed-f552-4edf-ac20-82310dd6cf44-data-volume\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.064701 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.064678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz7k\" (UniqueName: \"kubernetes.io/projected/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-api-access-pjz7k\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.266711 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.266666 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:19.327806 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.327772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:19.330751 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.330731 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 15:47:19.330860 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.330770 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf4f8\"" Jul 29 15:47:19.331827 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.331813 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 15:47:19.351960 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.351943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:19.352091 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.352076 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:19.352140 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.352131 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls podName:6cb7b2da-358a-4963-8d3c-0d31f5db64bb nodeName:}" failed. No retries permitted until 2026-07-29 15:47:21.352114881 +0000 UTC m=+37.590672072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls") pod "dns-default-rwjjb" (UID: "6cb7b2da-358a-4963-8d3c-0d31f5db64bb") : secret "dns-default-metrics-tls" not found Jul 29 15:47:19.375682 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.375664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:19.497544 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.497519 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="78bda61431733022b771393112f292510bab625ad542d4e3335be902ac7ef6cc" exitCode=0 Jul 29 15:47:19.497672 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.497601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"78bda61431733022b771393112f292510bab625ad542d4e3335be902ac7ef6cc"} Jul 29 15:47:19.498693 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.498671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vwgt7" event={"ID":"750ea16e-e156-4180-b5f8-067086f46f82","Type":"ContainerStarted","Data":"e11a52b364c2732ee9ed68265163eb40e4e73f9450c54da484426fbc89c9ddd3"} Jul 29 15:47:19.553799 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.553776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.553885 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.553815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:19.553932 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.553901 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:19.553971 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.553927 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.553971 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.553953 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.553939686 +0000 UTC m=+36.792496874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : configmap "kube-rbac-proxy" not found Jul 29 15:47:19.554065 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:19.553990 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:20.553961637 +0000 UTC m=+36.792518825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:19.557870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.557849 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-72mct"] Jul 29 15:47:19.560309 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.560296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" Jul 29 15:47:19.562926 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.562913 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 15:47:19.563226 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.563211 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 15:47:19.566221 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.563877 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6v4r2\"" Jul 29 15:47:19.576326 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.576307 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-72mct"] Jul 29 15:47:19.614129 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.614108 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:19.655461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.655426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhl9\" (UniqueName: \"kubernetes.io/projected/9a4196b4-9930-446b-88b1-ebadbd992e83-kube-api-access-pwhl9\") pod \"migrator-6865c9967-72mct\" (UID: \"9a4196b4-9930-446b-88b1-ebadbd992e83\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" Jul 29 15:47:19.708042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.708003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:19.756281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.756253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhl9\" (UniqueName: \"kubernetes.io/projected/9a4196b4-9930-446b-88b1-ebadbd992e83-kube-api-access-pwhl9\") pod \"migrator-6865c9967-72mct\" (UID: \"9a4196b4-9930-446b-88b1-ebadbd992e83\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" Jul 29 15:47:19.766132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.766102 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhl9\" (UniqueName: \"kubernetes.io/projected/9a4196b4-9930-446b-88b1-ebadbd992e83-kube-api-access-pwhl9\") pod \"migrator-6865c9967-72mct\" (UID: \"9a4196b4-9930-446b-88b1-ebadbd992e83\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" Jul 29 15:47:19.875845 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:19.875771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" Jul 29 15:47:20.014554 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.014525 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-72mct"] Jul 29 15:47:20.017342 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:20.017309 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4196b4_9930_446b_88b1_ebadbd992e83.slice/crio-e1036c6233fc4c84001a44bf8177f66fc8f7bd90ca5c15623a7f7168e887b5b1 WatchSource:0}: Error finding container e1036c6233fc4c84001a44bf8177f66fc8f7bd90ca5c15623a7f7168e887b5b1: Status 404 returned error can't find the container with id e1036c6233fc4c84001a44bf8177f66fc8f7bd90ca5c15623a7f7168e887b5b1 Jul 29 15:47:20.505048 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.505010 2574 generic.go:356] "Generic (PLEG): container finished" podID="648d11ed-cc61-426a-913f-cde108d6b63c" containerID="c0457dbcc66461e17831c7325221d02cbcda8d916b2a49151992aa87402dde38" exitCode=0 Jul 29 15:47:20.505539 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.505145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerDied","Data":"c0457dbcc66461e17831c7325221d02cbcda8d916b2a49151992aa87402dde38"} Jul 29 15:47:20.506935 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.506908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" event={"ID":"9a4196b4-9930-446b-88b1-ebadbd992e83","Type":"ContainerStarted","Data":"e1036c6233fc4c84001a44bf8177f66fc8f7bd90ca5c15623a7f7168e887b5b1"} Jul 29 15:47:20.563265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.563217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:20.563329 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.563281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:20.563362 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:20.563348 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:20.563412 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:20.563370 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 15:47:20.563456 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:20.563425 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:22.563405598 +0000 UTC m=+38.801962787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : configmap "kube-rbac-proxy" not found Jul 29 15:47:20.563456 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:20.563450 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:22.563440497 +0000 UTC m=+38.801997689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:20.571673 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.571650 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-994kz_af4961fd-5883-4b04-8eb0-1c950898d3ab/dns-node-resolver/0.log" Jul 29 15:47:20.760131 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.759947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:20.854073 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:20.854041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:21.370008 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:21.369958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:21.370169 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:21.370093 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:21.370169 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:21.370157 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls podName:6cb7b2da-358a-4963-8d3c-0d31f5db64bb nodeName:}" failed. No retries permitted until 2026-07-29 15:47:25.370141639 +0000 UTC m=+41.608698827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls") pod "dns-default-rwjjb" (UID: "6cb7b2da-358a-4963-8d3c-0d31f5db64bb") : secret "dns-default-metrics-tls" not found Jul 29 15:47:21.513040 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:21.513007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" event={"ID":"648d11ed-cc61-426a-913f-cde108d6b63c","Type":"ContainerStarted","Data":"6cde8173d7f6469ad9fecb9e58e355caac32f001fffacbbf8fa16e6057ed7c5d"} Jul 29 15:47:21.770411 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:21.770377 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:21.771480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:21.771465 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ptlz_2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4/node-ca/0.log" Jul 29 15:47:22.463260 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.463230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:22.516730 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.516695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vwgt7" event={"ID":"750ea16e-e156-4180-b5f8-067086f46f82","Type":"ContainerStarted","Data":"a55b116ff542aafdf2fe1d6c23f30b0f7939c4461f2c5477d1c462d77c27f9c0"} Jul 29 15:47:22.518511 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.518482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" event={"ID":"9a4196b4-9930-446b-88b1-ebadbd992e83","Type":"ContainerStarted","Data":"81c14e44dcbdb5cee6c3ef02e8d157a9b49fed2ca684c3c36c6b2bc57f4ac9c2"} Jul 29 15:47:22.518630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.518536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" event={"ID":"9a4196b4-9930-446b-88b1-ebadbd992e83","Type":"ContainerStarted","Data":"057013d6191476afd6111867f14018dcbc6eca45197810863e46e7701af06ca4"} Jul 29 15:47:22.537374 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.537334 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s5cv4" podStartSLOduration=9.014582791 podStartE2EDuration="38.537319215s" totalImagesPullingTime="29.522736424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:21.547239603 +0000 UTC m=+37.785796813" watchObservedRunningTime="2026-07-29 15:47:22.537319215 +0000 UTC m=+38.775876408" Jul 29 15:47:22.537646 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.537624 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vwgt7" podStartSLOduration=2.29190405 podStartE2EDuration="5.537615185s" totalImagesPullingTime="3.245711135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:22.5370727 +0000 UTC m=+38.775629922" watchObservedRunningTime="2026-07-29 15:47:22.537615185 +0000 UTC m=+38.776172394" Jul 29 15:47:22.570079 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.570031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:22.580144 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.580125 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:22.580233 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.580173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:22.580291 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:22.580268 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:22.580334 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:22.580324 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:26.580307501 +0000 UTC m=+42.818864706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:22.580724 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.580709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d5aebed-f552-4edf-ac20-82310dd6cf44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:22.774403 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:22.774332 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vwgt7_750ea16e-e156-4180-b5f8-067086f46f82/serve-healthcheck-canary/0.log" Jul 29 15:47:24.870598 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:24.870554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:25.013631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.013598 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:25.114238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.114204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:25.315197 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.315164 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:25.401834 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.401803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:25.402013 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:25.401903 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 15:47:25.402013 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:25.401955 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls podName:6cb7b2da-358a-4963-8d3c-0d31f5db64bb nodeName:}" failed. No retries permitted until 2026-07-29 15:47:33.401941689 +0000 UTC m=+49.640498877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls") pod "dns-default-rwjjb" (UID: "6cb7b2da-358a-4963-8d3c-0d31f5db64bb") : secret "dns-default-metrics-tls" not found Jul 29 15:47:25.469211 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.469171 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:25.973930 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:25.973888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:26.077457 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:26.077409 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:26.381062 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:26.380969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:26.573206 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:26.573173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:26.610527 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:26.610499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:26.610681 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:26.610615 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 15:47:26.610681 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:26.610671 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls podName:0d5aebed-f552-4edf-ac20-82310dd6cf44 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:34.610655206 +0000 UTC m=+50.849212408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4h77" (UID: "0d5aebed-f552-4edf-ac20-82310dd6cf44") : secret "insights-runtime-extractor-tls" not found Jul 29 15:47:27.162186 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:27.162151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:27.653739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:27.653697 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:27.759766 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:27.759735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:28.364692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:28.364664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:28.565696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:28.565657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:28.753543 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:28.753503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:28.822595 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:28.822550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:28.966780 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:28.966753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:29.368358 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:29.368322 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:29.468573 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:29.468536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:29.960349 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:29.960313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:30.358391 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:30.358286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:30.935196 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:30.935160 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:31.211864 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:31.211829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:31.289407 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:31.289368 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:31.578723 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:31.578645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:31.962114 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:31.962085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:32.434330 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:32.434242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:32.578941 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:32.578903 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:32.974360 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:32.974330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:33.371275 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.371185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:33.438694 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.438667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:33.456835 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.456800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:33.460331 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.460306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6cb7b2da-358a-4963-8d3c-0d31f5db64bb-metrics-tls\") pod \"dns-default-rwjjb\" (UID: \"6cb7b2da-358a-4963-8d3c-0d31f5db64bb\") " pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:33.565578 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.565543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:33.702459 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.702399 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-72mct" podStartSLOduration=12.806785726 podStartE2EDuration="14.702377447s" totalImagesPullingTime="1.895591721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:22.5671116 +0000 UTC m=+38.805668809" watchObservedRunningTime="2026-07-29 15:47:33.702377447 +0000 UTC m=+49.940934657" Jul 29 15:47:33.702946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:33.702908 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwjjb"] Jul 29 15:47:33.706178 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:33.706150 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb7b2da_358a_4963_8d3c_0d31f5db64bb.slice/crio-0616a5e6f4c2f4071973e9d6a7c16f5882d24b1197a22ec30195c87a6e13a2da WatchSource:0}: Error finding container 0616a5e6f4c2f4071973e9d6a7c16f5882d24b1197a22ec30195c87a6e13a2da: Status 404 returned error can't find the container with id 0616a5e6f4c2f4071973e9d6a7c16f5882d24b1197a22ec30195c87a6e13a2da Jul 29 15:47:34.521381 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.521343 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:34.549768 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.549729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwjjb" event={"ID":"6cb7b2da-358a-4963-8d3c-0d31f5db64bb","Type":"ContainerStarted","Data":"0616a5e6f4c2f4071973e9d6a7c16f5882d24b1197a22ec30195c87a6e13a2da"} Jul 29 15:47:34.621681 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.621644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:34.665532 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.665501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:34.667693 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.667676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d5aebed-f552-4edf-ac20-82310dd6cf44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4h77\" (UID: \"0d5aebed-f552-4edf-ac20-82310dd6cf44\") " pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:34.689724 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.689691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4h77" Jul 29 15:47:34.780349 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.780228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:34.854491 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:34.854447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4h77"] Jul 29 15:47:35.167478 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.167373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:35.247862 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:35.247826 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5aebed_f552_4edf_ac20_82310dd6cf44.slice/crio-b2dcbce0afab955415290e36096a648fde3f3de06025c1feeedb2a11c05742bf WatchSource:0}: Error finding container b2dcbce0afab955415290e36096a648fde3f3de06025c1feeedb2a11c05742bf: Status 404 returned error can't find the container with id b2dcbce0afab955415290e36096a648fde3f3de06025c1feeedb2a11c05742bf Jul 29 15:47:35.359183 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.359150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:35.553260 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.553232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwjjb" event={"ID":"6cb7b2da-358a-4963-8d3c-0d31f5db64bb","Type":"ContainerStarted","Data":"60c3d370417a92cfc9d07dce1a7978b2c1de33324185d66b56e31152875a7bbf"} Jul 29 15:47:35.554251 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.554230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4h77" event={"ID":"0d5aebed-f552-4edf-ac20-82310dd6cf44","Type":"ContainerStarted","Data":"b2dcbce0afab955415290e36096a648fde3f3de06025c1feeedb2a11c05742bf"} Jul 29 15:47:35.571054 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.571022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:35.671729 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:35.671685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:36.560718 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:36.560678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwjjb" event={"ID":"6cb7b2da-358a-4963-8d3c-0d31f5db64bb","Type":"ContainerStarted","Data":"581d9798cf936102e1cad75b2b8858e1c84c7c31b81b6f5d8e61d7e6874ea99a"} Jul 29 15:47:36.561214 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:36.560841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:36.594376 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:36.594314 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rwjjb" podStartSLOduration=17.919960014 podStartE2EDuration="19.594294182s" totalImagesPullingTime="1.674334168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:36.592873746 +0000 UTC m=+52.831430955" watchObservedRunningTime="2026-07-29 15:47:36.594294182 +0000 UTC m=+52.832851395" Jul 29 15:47:40.578504 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:40.575856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4h77" event={"ID":"0d5aebed-f552-4edf-ac20-82310dd6cf44","Type":"ContainerStarted","Data":"5a4396a7de03b236235cf4cfcfed4be9a02037b17f55fade494f1754b4c001d3"} Jul 29 15:47:43.497092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:43.497059 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r4hgq" Jul 29 15:47:43.744511 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:43.744452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:43.844924 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:43.844816 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:43.993303 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:43.993261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.184995 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.184899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.377383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.377351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.472700 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.472677 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.572472 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.572444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.591526 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.591497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4h77" event={"ID":"0d5aebed-f552-4edf-ac20-82310dd6cf44","Type":"ContainerStarted","Data":"b528758eb59c68ec8fc1f43a9d9bdc93a31caf640581cd4e2cb98e72fd06baa3"} Jul 29 15:47:44.636235 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.636205 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.771057 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.770958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:44.844114 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:44.844071 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.160060 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.159989 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6"] Jul 29 15:47:45.163016 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.162939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.167037 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.166844 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 15:47:45.167303 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.167277 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 15:47:45.168318 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.168221 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 15:47:45.170353 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.170331 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 15:47:45.170442 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.170397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 15:47:45.170560 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.170542 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 15:47:45.170616 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.170587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 15:47:45.249377 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.249539 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.249539 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlgs\" (UniqueName: \"kubernetes.io/projected/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-kube-api-access-jvlgs\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.249539 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.249668 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.249668 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.249584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.289884 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.289845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6"] Jul 29 15:47:45.350210 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlgs\" (UniqueName: \"kubernetes.io/projected/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-kube-api-access-jvlgs\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.350396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.350396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.350396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.350396 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.350659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.350625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.351395 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.351163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.353947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.353363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.353947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.353416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-ca\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.353947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.353476 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.353947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.353912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.378495 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.378465 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlgs\" (UniqueName: \"kubernetes.io/projected/b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17-kube-api-access-jvlgs\") pod \"cluster-proxy-proxy-agent-6549484c7d-g2ks6\" (UID: \"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.423029 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.422933 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-849bb45f95-6djs8"] Jul 29 15:47:45.425533 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.425507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.430781 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.430755 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgnpm\"" Jul 29 15:47:45.432692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.432650 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 15:47:45.432902 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.432884 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 15:47:45.434106 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.433860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 15:47:45.439245 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.439219 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 15:47:45.446858 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.446836 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849bb45f95-6djs8"] Jul 29 15:47:45.451801 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-bound-sa-token\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.451929 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-trusted-ca\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.451929 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-image-registry-private-configuration\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.451929 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-installation-pull-secrets\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.452112 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-tls\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.452112 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.451958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-certificates\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.452112 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.452012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30bd5209-f3e4-4e63-8378-f1330a3a821d-ca-trust-extracted\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.452112 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.452029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfst4\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-kube-api-access-bfst4\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.459460 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.459432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.481996 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.481941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" Jul 29 15:47:45.553961 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.553925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-image-registry-private-configuration\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554162 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.553971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-installation-pull-secrets\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554162 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-tls\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554162 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-certificates\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30bd5209-f3e4-4e63-8378-f1330a3a821d-ca-trust-extracted\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfst4\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-kube-api-access-bfst4\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-bound-sa-token\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-trusted-ca\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.554915 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.554883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30bd5209-f3e4-4e63-8378-f1330a3a821d-ca-trust-extracted\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.555445 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.555421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-trusted-ca\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.555916 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.555856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-certificates\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.558379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.558357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-image-registry-private-configuration\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.558807 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.558782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30bd5209-f3e4-4e63-8378-f1330a3a821d-installation-pull-secrets\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.559707 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.559687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-registry-tls\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.574234 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.574208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfst4\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-kube-api-access-bfst4\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.575592 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.575532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.578158 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.578134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30bd5209-f3e4-4e63-8378-f1330a3a821d-bound-sa-token\") pod \"image-registry-849bb45f95-6djs8\" (UID: \"30bd5209-f3e4-4e63-8378-f1330a3a821d\") " pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.637503 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.637353 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6"] Jul 29 15:47:45.640121 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:45.640085 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4b07e25_d8bc_433e_a0e8_3aee8f1b3b17.slice/crio-c481a7a816afa32847908e93b7d0cdf4cfb2af9b50028ddffa899264367109a3 WatchSource:0}: Error finding container c481a7a816afa32847908e93b7d0cdf4cfb2af9b50028ddffa899264367109a3: Status 404 returned error can't find the container with id c481a7a816afa32847908e93b7d0cdf4cfb2af9b50028ddffa899264367109a3 Jul 29 15:47:45.669481 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.669454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.737790 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.737763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:45.769386 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.769358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.872386 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.872348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:45.884260 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:45.884228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849bb45f95-6djs8"] Jul 29 15:47:45.887113 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:45.887081 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bd5209_f3e4_4e63_8378_f1330a3a821d.slice/crio-475575e65771ddb41a575b1142228c1b9ca668451fdade7fa111cddefc9c9c78 WatchSource:0}: Error finding container 475575e65771ddb41a575b1142228c1b9ca668451fdade7fa111cddefc9c9c78: Status 404 returned error can't find the container with id 475575e65771ddb41a575b1142228c1b9ca668451fdade7fa111cddefc9c9c78 Jul 29 15:47:46.110581 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.110430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:46.403165 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.403056 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:46.566562 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.566534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwjjb" Jul 29 15:47:46.602835 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.602687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" event={"ID":"30bd5209-f3e4-4e63-8378-f1330a3a821d","Type":"ContainerStarted","Data":"4e5c7af27947469747fc72881d8c29d9362333ec133d438aecc9798422f0367f"} Jul 29 15:47:46.602835 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.602756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" event={"ID":"30bd5209-f3e4-4e63-8378-f1330a3a821d","Type":"ContainerStarted","Data":"475575e65771ddb41a575b1142228c1b9ca668451fdade7fa111cddefc9c9c78"} Jul 29 15:47:46.602835 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.602804 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:47:46.604322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.604270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" event={"ID":"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17","Type":"ContainerStarted","Data":"c481a7a816afa32847908e93b7d0cdf4cfb2af9b50028ddffa899264367109a3"} Jul 29 15:47:46.626738 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:46.626675 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" podStartSLOduration=1.6266536390000002 podStartE2EDuration="1.626653639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:46.625554827 +0000 UTC m=+62.864112038" watchObservedRunningTime="2026-07-29 15:47:46.626653639 +0000 UTC m=+62.865210844" Jul 29 15:47:47.536127 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:47.536089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:47.638471 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:47.638435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:47.736885 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:47.736847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:47.821732 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:47.821709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:48.609726 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.609645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:48.614160 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.614085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4h77" event={"ID":"0d5aebed-f552-4edf-ac20-82310dd6cf44","Type":"ContainerStarted","Data":"ed18fb4c70df2ab92b1b68397a7a9765dc5ddc9ef9ff593aa08f000cc9b81c6f"} Jul 29 15:47:48.634808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.634739 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v4h77" podStartSLOduration=18.198020617 podStartE2EDuration="30.634717829s" totalImagesPullingTime="12.436697212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:47:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:48.632928068 +0000 UTC m=+64.871485282" watchObservedRunningTime="2026-07-29 15:47:48.634717829 +0000 UTC m=+64.873275040" Jul 29 15:47:48.982768 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.982725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:48.985848 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.985820 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 15:47:48.996322 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:48.996292 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 15:47:49.007780 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.007753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6k4\" (UniqueName: \"kubernetes.io/projected/f48a0452-0f23-4911-aa5f-dddc65ea5d5e-kube-api-access-rj6k4\") pod \"network-check-target-7224n\" (UID: \"f48a0452-0f23-4911-aa5f-dddc65ea5d5e\") " pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:49.039309 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.039272 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf4f8\"" Jul 29 15:47:49.046793 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.046765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:49.083467 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.083432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:49.083655 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.083514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:49.086610 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.086580 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 15:47:49.086724 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.086616 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 15:47:49.097365 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.097336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/901c05b2-86a2-4a19-9e37-99b5f4b79c26-original-pull-secret\") pod \"global-pull-secret-syncer-w9k2g\" (UID: \"901c05b2-86a2-4a19-9e37-99b5f4b79c26\") " pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:49.097365 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.097357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/839cb74e-cea4-4bfb-99fb-010af6495d0b-metrics-certs\") pod \"network-metrics-daemon-jmxs9\" (UID: \"839cb74e-cea4-4bfb-99fb-010af6495d0b\") " pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:49.239472 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.239396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9k2g" Jul 29 15:47:49.247989 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.247953 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28zhr\"" Jul 29 15:47:49.255241 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.255215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmxs9" Jul 29 15:47:49.470871 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.470798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmxs9"] Jul 29 15:47:49.474692 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:49.474661 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839cb74e_cea4_4bfb_99fb_010af6495d0b.slice/crio-97a0b3a0ffe213e1c3be85ed2679e5d12dfcaff9247d73ac3d183483cbc954f7 WatchSource:0}: Error finding container 97a0b3a0ffe213e1c3be85ed2679e5d12dfcaff9247d73ac3d183483cbc954f7: Status 404 returned error can't find the container with id 97a0b3a0ffe213e1c3be85ed2679e5d12dfcaff9247d73ac3d183483cbc954f7 Jul 29 15:47:49.480392 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.480352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:49.491109 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.491044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7224n"] Jul 29 15:47:49.494306 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:49.494281 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48a0452_0f23_4911_aa5f_dddc65ea5d5e.slice/crio-83fd9e9b536355ee8056f37f28e31ed409c446117f3504694a9c39189571d96c WatchSource:0}: Error finding container 83fd9e9b536355ee8056f37f28e31ed409c446117f3504694a9c39189571d96c: Status 404 returned error can't find the container with id 83fd9e9b536355ee8056f37f28e31ed409c446117f3504694a9c39189571d96c Jul 29 15:47:49.571150 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.571112 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:49.618554 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.618507 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7224n" event={"ID":"f48a0452-0f23-4911-aa5f-dddc65ea5d5e","Type":"ContainerStarted","Data":"83fd9e9b536355ee8056f37f28e31ed409c446117f3504694a9c39189571d96c"} Jul 29 15:47:49.619800 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.619773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmxs9" event={"ID":"839cb74e-cea4-4bfb-99fb-010af6495d0b","Type":"ContainerStarted","Data":"97a0b3a0ffe213e1c3be85ed2679e5d12dfcaff9247d73ac3d183483cbc954f7"} Jul 29 15:47:49.692904 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:49.692862 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9k2g"] Jul 29 15:47:50.602935 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:50.602894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:50.667295 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:50.667259 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901c05b2_86a2_4a19_9e37_99b5f4b79c26.slice/crio-f7fca54de181c0f2976b86040f56719c0638ba461a5c2528ecc65d218ab457de WatchSource:0}: Error finding container f7fca54de181c0f2976b86040f56719c0638ba461a5c2528ecc65d218ab457de: Status 404 returned error can't find the container with id f7fca54de181c0f2976b86040f56719c0638ba461a5c2528ecc65d218ab457de Jul 29 15:47:50.718250 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:50.718213 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:51.628643 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:51.628605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" event={"ID":"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17","Type":"ContainerStarted","Data":"fb94b614c5c3482ace135159eb078cc26fd37f7ba8d266b89ec9fba45f6395f7"} Jul 29 15:47:51.629072 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:51.628662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" event={"ID":"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17","Type":"ContainerStarted","Data":"495f4a5688dc5e5e0c736cda5d5caafb335e695cdb09cee4a68acb0da0d2dedb"} Jul 29 15:47:51.629970 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:51.629935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9k2g" event={"ID":"901c05b2-86a2-4a19-9e37-99b5f4b79c26","Type":"ContainerStarted","Data":"f7fca54de181c0f2976b86040f56719c0638ba461a5c2528ecc65d218ab457de"} Jul 29 15:47:51.658290 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:51.658146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:51.922963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:51.922930 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:52.211351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.211313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:52.636895 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.636560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmxs9" event={"ID":"839cb74e-cea4-4bfb-99fb-010af6495d0b","Type":"ContainerStarted","Data":"5495d736ef689d0368ca45cfb74648a4bcb7a3711047bdb00ec9537837e40028"} Jul 29 15:47:52.636895 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.636633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmxs9" event={"ID":"839cb74e-cea4-4bfb-99fb-010af6495d0b","Type":"ContainerStarted","Data":"a962db17bcc67db785a1b2e94ee8f2ec8907b6682387b669d8a40c83b7098d85"} Jul 29 15:47:52.639414 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.639385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" event={"ID":"b4b07e25-d8bc-433e-a0e8-3aee8f1b3b17","Type":"ContainerStarted","Data":"02345ca1b5021d4adc2872fc92d65bce311a33100ce8c01e12272926c7088688"} Jul 29 15:47:52.655354 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.655062 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jmxs9" podStartSLOduration=66.221053441 podStartE2EDuration="1m8.655042278s" totalImagesPullingTime="2.433988837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:52.653355445 +0000 UTC m=+68.891912656" watchObservedRunningTime="2026-07-29 15:47:52.655042278 +0000 UTC m=+68.893599489" Jul 29 15:47:52.764448 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.764408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:52.828332 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:52.828293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:54.762863 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:54.762809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:54.864772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:54.864719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:55.651619 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.651577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9k2g" event={"ID":"901c05b2-86a2-4a19-9e37-99b5f4b79c26","Type":"ContainerStarted","Data":"3a64d1ed4c868103e324aad70179a9760760c8319908e0c222208634fa0689ad"} Jul 29 15:47:55.653045 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.653019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7224n" event={"ID":"f48a0452-0f23-4911-aa5f-dddc65ea5d5e","Type":"ContainerStarted","Data":"907bd1e66edc0f6e1bbcc179b55d8b19433753f8668571326a675bf6437aa142"} Jul 29 15:47:55.653169 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.653084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:47:55.670740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.670692 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549484c7d-g2ks6" podStartSLOduration=5.326443699 podStartE2EDuration="10.670677399s" totalImagesPullingTime="5.3442337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:52.676570042 +0000 UTC m=+68.915127252" watchObservedRunningTime="2026-07-29 15:47:55.670677399 +0000 UTC m=+71.909234613" Jul 29 15:47:55.671336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.671309 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w9k2g" podStartSLOduration=67.366294416 podStartE2EDuration="1m11.671299987s" totalImagesPullingTime="4.305005571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:55.670461657 +0000 UTC m=+71.909018868" watchObservedRunningTime="2026-07-29 15:47:55.671299987 +0000 UTC m=+71.909857175" Jul 29 15:47:55.687824 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.687777 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7224n" podStartSLOduration=66.213298765 podStartE2EDuration="1m11.687763582s" totalImagesPullingTime="5.474464817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:47:55.686504683 +0000 UTC m=+71.925061895" watchObservedRunningTime="2026-07-29 15:47:55.687763582 +0000 UTC m=+71.926320791" Jul 29 15:47:55.732672 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.732644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:55.834104 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:55.834055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:57.783282 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.783246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:57.816999 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.816958 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c245l"] Jul 29 15:47:57.819227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.819211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.821882 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.821860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 29 15:47:57.823324 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.823308 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 15:47:57.823429 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.823381 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 15:47:57.823497 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.823427 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 29 15:47:57.823497 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.823460 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-55gmg\"" Jul 29 15:47:57.823602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.823567 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 15:47:57.830281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.830258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c245l"] Jul 29 15:47:57.856557 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.856531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.856557 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.856558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8qd\" (UniqueName: \"kubernetes.io/projected/06699a36-83be-48d8-b653-9d549d117cb5-kube-api-access-8k8qd\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.856694 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.856587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06699a36-83be-48d8-b653-9d549d117cb5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.856694 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.856643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.857340 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.857323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:57.957828 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.957803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.957905 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.957851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.957905 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.957875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8qd\" (UniqueName: \"kubernetes.io/projected/06699a36-83be-48d8-b653-9d549d117cb5-kube-api-access-8k8qd\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.957965 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.957917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06699a36-83be-48d8-b653-9d549d117cb5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.958023 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:57.957971 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 29 15:47:57.958059 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:47:57.958053 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls podName:06699a36-83be-48d8-b653-9d549d117cb5 nodeName:}" failed. No retries permitted until 2026-07-29 15:47:58.458033027 +0000 UTC m=+74.696590232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-c245l" (UID: "06699a36-83be-48d8-b653-9d549d117cb5") : secret "prometheus-operator-tls" not found Jul 29 15:47:57.958587 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.958568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06699a36-83be-48d8-b653-9d549d117cb5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.960274 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.960254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:57.966735 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:57.966714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8qd\" (UniqueName: \"kubernetes.io/projected/06699a36-83be-48d8-b653-9d549d117cb5-kube-api-access-8k8qd\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:58.096707 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:58.096628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:58.462825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:58.462799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:58.465156 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:58.465138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/06699a36-83be-48d8-b653-9d549d117cb5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-c245l\" (UID: \"06699a36-83be-48d8-b653-9d549d117cb5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:58.727996 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:58.727887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" Jul 29 15:47:58.851412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:58.851389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-c245l"] Jul 29 15:47:58.853732 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:47:58.853708 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06699a36_83be_48d8_b653_9d549d117cb5.slice/crio-ba1dcda3ab05fabe7a639886ee5b41c85c248c00372721b79acc95f359a6fd8a WatchSource:0}: Error finding container ba1dcda3ab05fabe7a639886ee5b41c85c248c00372721b79acc95f359a6fd8a: Status 404 returned error can't find the container with id ba1dcda3ab05fabe7a639886ee5b41c85c248c00372721b79acc95f359a6fd8a Jul 29 15:47:59.199901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:59.199815 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:59.452678 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:59.452596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:59.522467 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:59.522433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:47:59.667607 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:47:59.667577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" event={"ID":"06699a36-83be-48d8-b653-9d549d117cb5","Type":"ContainerStarted","Data":"ba1dcda3ab05fabe7a639886ee5b41c85c248c00372721b79acc95f359a6fd8a"} Jul 29 15:48:00.672917 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:00.672834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" event={"ID":"06699a36-83be-48d8-b653-9d549d117cb5","Type":"ContainerStarted","Data":"e4fcc58c17bb8e1a15448e0f6677fde28288e36a6bd566115242fc5f011ca32a"} Jul 29 15:48:00.672917 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:00.672900 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" event={"ID":"06699a36-83be-48d8-b653-9d549d117cb5","Type":"ContainerStarted","Data":"ded1cd1fb87daa765dfa3737edd938dde5849bfb12fa9398710906d6b095a9f7"} Jul 29 15:48:00.691371 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:00.691207 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-c245l" podStartSLOduration=2.126857394 podStartE2EDuration="3.691162379s" totalImagesPullingTime="1.564304985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:00.689768768 +0000 UTC m=+76.928325996" watchObservedRunningTime="2026-07-29 15:48:00.691162379 +0000 UTC m=+76.929719589" Jul 29 15:48:01.079298 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:01.079256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:01.145571 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:01.145545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:01.924208 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:01.924170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:02.148748 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.148714 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:02.200809 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.200729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfpwj"] Jul 29 15:48:02.203202 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.203186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.205969 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.205944 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 15:48:02.205969 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.205966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 15:48:02.206179 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.205966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hcxj9\"" Jul 29 15:48:02.206179 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.206035 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 15:48:02.260128 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.260081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:02.289115 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289115 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-sys\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-wtmp\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-metrics-client-ca\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289337 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-textfile\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnzs\" (UniqueName: \"kubernetes.io/projected/f497917b-1860-4d56-ac85-debb1e9b6e07-kube-api-access-flnzs\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.289506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.289395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-root\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.350651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.350619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:02.390207 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnzs\" (UniqueName: \"kubernetes.io/projected/f497917b-1860-4d56-ac85-debb1e9b6e07-kube-api-access-flnzs\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-root\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-root\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-sys\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390338 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-sys\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-wtmp\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-metrics-client-ca\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-textfile\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:48:02.390402 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-wtmp\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390617 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:48:02.390534 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls podName:f497917b-1860-4d56-ac85-debb1e9b6e07 nodeName:}" failed. No retries permitted until 2026-07-29 15:48:02.89051263 +0000 UTC m=+79.129069839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls") pod "node-exporter-pfpwj" (UID: "f497917b-1860-4d56-ac85-debb1e9b6e07") : secret "node-exporter-tls" not found Jul 29 15:48:02.390822 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-textfile\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.390944 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.390911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.391087 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.391068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f497917b-1860-4d56-ac85-debb1e9b6e07-metrics-client-ca\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.392673 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.392658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.403540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.403519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnzs\" (UniqueName: \"kubernetes.io/projected/f497917b-1860-4d56-ac85-debb1e9b6e07-kube-api-access-flnzs\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.493901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.493856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:02.894701 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.894616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.896854 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.896832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f497917b-1860-4d56-ac85-debb1e9b6e07-node-exporter-tls\") pod \"node-exporter-pfpwj\" (UID: \"f497917b-1860-4d56-ac85-debb1e9b6e07\") " pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:02.934087 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:02.934064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.035260 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.035228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.112600 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.112572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfpwj" Jul 29 15:48:03.122259 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:48:03.122235 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf497917b_1860_4d56_ac85_debb1e9b6e07.slice/crio-4957ff9ab43f7a5b0299b34b99946510ee7cee4b02a5cf34f1a17b5ac15fd998 WatchSource:0}: Error finding container 4957ff9ab43f7a5b0299b34b99946510ee7cee4b02a5cf34f1a17b5ac15fd998: Status 404 returned error can't find the container with id 4957ff9ab43f7a5b0299b34b99946510ee7cee4b02a5cf34f1a17b5ac15fd998 Jul 29 15:48:03.205726 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.205699 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.309591 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.309554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.508536 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.508453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.600687 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.600650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:03.682999 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.682957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfpwj" event={"ID":"f497917b-1860-4d56-ac85-debb1e9b6e07","Type":"ContainerStarted","Data":"4957ff9ab43f7a5b0299b34b99946510ee7cee4b02a5cf34f1a17b5ac15fd998"} Jul 29 15:48:03.762255 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:03.762180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:04.688403 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:04.688312 2574 generic.go:356] "Generic (PLEG): container finished" podID="f497917b-1860-4d56-ac85-debb1e9b6e07" containerID="1d25672c18de77776e2826c6e0e6fc6d1699631974d41985ca38ed44a2425aa3" exitCode=0 Jul 29 15:48:04.688403 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:04.688363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfpwj" event={"ID":"f497917b-1860-4d56-ac85-debb1e9b6e07","Type":"ContainerDied","Data":"1d25672c18de77776e2826c6e0e6fc6d1699631974d41985ca38ed44a2425aa3"} Jul 29 15:48:04.766963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:04.766934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:04.866854 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:04.866824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:05.182808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.182774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:05.333677 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.333645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:05.578164 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.578044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:05.678846 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.678810 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:05.693594 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.693564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfpwj" event={"ID":"f497917b-1860-4d56-ac85-debb1e9b6e07","Type":"ContainerStarted","Data":"22aa7d8f3aca622b8f91263bb0254e9686af4c0523ba99c64b42246930c8b335"} Jul 29 15:48:05.693946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.693640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfpwj" event={"ID":"f497917b-1860-4d56-ac85-debb1e9b6e07","Type":"ContainerStarted","Data":"3c915219791c5959bca68f48a6425cb995e275f05a60681befd53002c8ff1df0"} Jul 29 15:48:05.723386 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:05.723336 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfpwj" podStartSLOduration=2.36040454 podStartE2EDuration="3.723320225s" totalImagesPullingTime="1.298066647s" totalInitContainerRuntime="64.849038ms" isStatefulPod=true podCreationTimestamp="2026-07-29 15:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:05.722697877 +0000 UTC m=+81.961255086" watchObservedRunningTime="2026-07-29 15:48:05.723320225 +0000 UTC m=+81.961877436" Jul 29 15:48:06.178776 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.178742 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:06.957239 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.957205 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:06.970447 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.970418 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6"] Jul 29 15:48:06.973090 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.973072 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:06.976705 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.976688 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 29 15:48:06.976895 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.976877 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q2zdx\"" Jul 29 15:48:06.983770 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:06.983747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6"] Jul 29 15:48:07.024481 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.024449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz842\" (UniqueName: \"kubernetes.io/projected/e94fa930-5921-4a69-bab2-273ff8122b6b-kube-api-access-fz842\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.024660 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.024520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94fa930-5921-4a69-bab2-273ff8122b6b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.125535 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.125499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94fa930-5921-4a69-bab2-273ff8122b6b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.125707 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.125546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz842\" (UniqueName: \"kubernetes.io/projected/e94fa930-5921-4a69-bab2-273ff8122b6b-kube-api-access-fz842\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.127937 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.127918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94fa930-5921-4a69-bab2-273ff8122b6b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.134438 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.134407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz842\" (UniqueName: \"kubernetes.io/projected/e94fa930-5921-4a69-bab2-273ff8122b6b-kube-api-access-fz842\") pod \"monitoring-plugin-68b7fc9574-lzwd6\" (UID: \"e94fa930-5921-4a69-bab2-273ff8122b6b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.282418 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.282328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:07.408371 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.408346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6"] Jul 29 15:48:07.410633 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:48:07.410611 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94fa930_5921_4a69_bab2_273ff8122b6b.slice/crio-5bd8de40df33bd3ef297aabb46b314f5cb0809204fec3e39bf623412a8c149fa WatchSource:0}: Error finding container 5bd8de40df33bd3ef297aabb46b314f5cb0809204fec3e39bf623412a8c149fa: Status 404 returned error can't find the container with id 5bd8de40df33bd3ef297aabb46b314f5cb0809204fec3e39bf623412a8c149fa Jul 29 15:48:07.509559 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.509526 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:07.612736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.612660 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-849bb45f95-6djs8" Jul 29 15:48:07.652181 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.652151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:07.701505 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.701476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" event={"ID":"e94fa930-5921-4a69-bab2-273ff8122b6b","Type":"ContainerStarted","Data":"5bd8de40df33bd3ef297aabb46b314f5cb0809204fec3e39bf623412a8c149fa"} Jul 29 15:48:07.764425 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.764393 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:07.875576 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:07.875497 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:08.476791 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.476743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:08.483627 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.483599 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:48:08.491892 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.491864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.495618 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.495595 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 15:48:08.495795 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.495777 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 15:48:08.496249 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.496231 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 15:48:08.496520 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.496501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 15:48:08.497085 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497067 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 15:48:08.497165 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497094 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fnwgf\"" Jul 29 15:48:08.497165 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497141 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 15:48:08.497267 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497210 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bupb5r3gdhidh\"" Jul 29 15:48:08.497579 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497565 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 15:48:08.497749 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497736 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 15:48:08.497809 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 15:48:08.497866 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497827 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 15:48:08.497917 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497870 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 15:48:08.497970 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 15:48:08.497970 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.497942 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 15:48:08.499503 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.499483 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 15:48:08.509630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.509610 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 15:48:08.515301 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.515273 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:48:08.637968 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.637927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.637992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638397 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhxl\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.638946 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.638674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.739861 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.739861 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.739861 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.739861 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhxl\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.739959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740215 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740645 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.740870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.741166 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.740971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.743188 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.742851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.743188 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.743078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.743188 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.743175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.743416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.743360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.743691 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.743666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.744207 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.744181 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.744764 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.744740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.745242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.745177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.745544 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.745524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.745949 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.745923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.746101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.746076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.746166 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.746142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.746943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.746302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.746943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.746458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.746943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.746893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.752097 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.752078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhxl\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl\") pod \"prometheus-k8s-0\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:08.805181 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:08.805138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:09.023121 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:09.023090 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:48:09.025821 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:48:09.025791 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf75ab9_cc9b_4334_bce2_a11f01e7e941.slice/crio-1e9574e9d77d4359bff8dbcccbaea75d5670ce7693f55f37f09e6a843ed01a53 WatchSource:0}: Error finding container 1e9574e9d77d4359bff8dbcccbaea75d5670ce7693f55f37f09e6a843ed01a53: Status 404 returned error can't find the container with id 1e9574e9d77d4359bff8dbcccbaea75d5670ce7693f55f37f09e6a843ed01a53 Jul 29 15:48:09.712436 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:09.712381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"1e9574e9d77d4359bff8dbcccbaea75d5670ce7693f55f37f09e6a843ed01a53"} Jul 29 15:48:10.219007 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.218897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:10.498035 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.497949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:10.717647 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.717616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" event={"ID":"e94fa930-5921-4a69-bab2-273ff8122b6b","Type":"ContainerStarted","Data":"66faca9916ccabc6c62f384870b18ed90852e235b86b2e4866f6caf3b018098a"} Jul 29 15:48:10.718098 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.717811 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:10.722769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.722745 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" Jul 29 15:48:10.734872 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:10.734826 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lzwd6" podStartSLOduration=1.604800029 podStartE2EDuration="4.734811539s" totalImagesPullingTime="3.13001151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:10.733815826 +0000 UTC m=+86.972373040" watchObservedRunningTime="2026-07-29 15:48:10.734811539 +0000 UTC m=+86.973368780" Jul 29 15:48:11.645635 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:11.645597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:11.714073 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:11.714045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:11.723182 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:11.723155 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" exitCode=0 Jul 29 15:48:11.723506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:11.723247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} Jul 29 15:48:13.783490 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:13.783447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:13.883930 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:13.883899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:14.185096 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:14.184972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:14.285480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:14.285437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:15.742909 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:15.742877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} Jul 29 15:48:15.743296 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:15.742946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} Jul 29 15:48:16.644521 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:16.644487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:17.159661 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.159628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:17.752841 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.752809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} Jul 29 15:48:17.753081 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.752914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} Jul 29 15:48:17.753081 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.752955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} Jul 29 15:48:17.753081 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.753018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerStarted","Data":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} Jul 29 15:48:17.786347 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:17.786283 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6901227629999998 podStartE2EDuration="9.786261801s" totalImagesPullingTime="7.082858173s" totalInitContainerRuntime="13.280865ms" isStatefulPod=false podCreationTimestamp="2026-07-29 15:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:48:17.784336987 +0000 UTC m=+94.022894213" watchObservedRunningTime="2026-07-29 15:48:17.786261801 +0000 UTC m=+94.024819011" Jul 29 15:48:18.805549 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:18.805510 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:48:20.602419 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:20.602381 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:20.738375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:20.738345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:23.715240 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:23.715199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:23.799903 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:23.799869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:24.450611 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:24.450581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vwgt7_750ea16e-e156-4180-b5f8-067086f46f82/serve-healthcheck-canary/0.log" Jul 29 15:48:26.659389 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:26.659355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7224n" Jul 29 15:48:30.899420 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:30.899383 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:30.999741 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:30.999712 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:31.060041 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:31.060002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:34.298758 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:34.298723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:34.363414 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:34.363386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:35.601193 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:35.601161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:48:35.669274 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:48:35.669242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:03.880815 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:03.880777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:08.805368 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:08.805322 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:08.824545 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:08.824519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:08.951584 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:08.951553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:18.343821 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:18.343776 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:18.438150 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:18.438120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:19.816304 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:19.816253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:26.887826 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.887782 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:26.888451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888195 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="prometheus" containerID="cri-o://0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" gracePeriod=600 Jul 29 15:49:26.888598 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888569 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-thanos" containerID="cri-o://43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" gracePeriod=600 Jul 29 15:49:26.888706 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888687 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy" containerID="cri-o://524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" gracePeriod=600 Jul 29 15:49:26.888802 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888776 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-web" containerID="cri-o://e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" gracePeriod=600 Jul 29 15:49:26.888949 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888897 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="config-reloader" containerID="cri-o://d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" gracePeriod=600 Jul 29 15:49:26.889046 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:26.888892 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="thanos-sidecar" containerID="cri-o://a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" gracePeriod=600 Jul 29 15:49:27.151875 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.151848 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:27.282637 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282602 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282637 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282637 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282661 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282690 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282723 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.282900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.282769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283175 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283027 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.283175 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283108 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.283265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283172 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283225 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283286 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxhxl\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283389 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283440 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283465 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283496 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283539 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283574 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283605 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0\") pod \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\" (UID: \"5bf75ab9-cc9b-4334-bce2-a11f01e7e941\") " Jul 29 15:49:27.283849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283628 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.283963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283845 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.283963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283864 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.283963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.283881 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.284415 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.284385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.284415 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.284397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.285547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.285514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.285775 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.285752 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config" (OuterVolumeSpecName: "config") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.286259 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.286233 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:49:27.286357 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.286316 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.286548 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.286518 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.286625 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.286586 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.286909 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.286876 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.287109 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.287082 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:49:27.287830 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.287787 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out" (OuterVolumeSpecName: "config-out") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:49:27.287926 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.287828 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.288019 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.287998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.288150 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.288129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.288336 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.288319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:49:27.289233 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.289214 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl" (OuterVolumeSpecName: "kube-api-access-xxhxl") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "kube-api-access-xxhxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:49:27.296140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.296123 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config" (OuterVolumeSpecName: "web-config") pod "5bf75ab9-cc9b-4334-bce2-a11f01e7e941" (UID: "5bf75ab9-cc9b-4334-bce2-a11f01e7e941"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:49:27.384232 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384197 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384232 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384227 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-tls-assets\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384242 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384255 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-metrics-client-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384270 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384283 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384294 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384306 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-web-config\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384319 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-grpc-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384329 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384340 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-metrics-client-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384350 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-config-out\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384362 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384373 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384386 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-prometheus-k8s-db\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384397 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhxl\" (UniqueName: \"kubernetes.io/projected/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-kube-api-access-xxhxl\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:27.384473 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:27.384410 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bf75ab9-cc9b-4334-bce2-a11f01e7e941-secret-kube-rbac-proxy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:49:28.006100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006070 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" exitCode=0 Jul 29 15:49:28.006100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006093 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" exitCode=0 Jul 29 15:49:28.006100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006100 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" exitCode=0 Jul 29 15:49:28.006100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006105 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" exitCode=0 Jul 29 15:49:28.006100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006112 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" exitCode=0 Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006117 2574 generic.go:356] "Generic (PLEG): container finished" podID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" exitCode=0 Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006159 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006356 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006394 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} Jul 29 15:49:28.006696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.006502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bf75ab9-cc9b-4334-bce2-a11f01e7e941","Type":"ContainerDied","Data":"1e9574e9d77d4359bff8dbcccbaea75d5670ce7693f55f37f09e6a843ed01a53"} Jul 29 15:49:28.014881 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.014859 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.021641 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.021625 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.028139 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.028121 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.031661 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.031637 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:28.036257 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.036227 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:28.036359 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.036233 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.042901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.042881 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.049903 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.049885 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.056437 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.056422 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.056700 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.056681 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.056744 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.056710 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.056789 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.056743 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.057100 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.056962 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.057100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057006 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.057100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057031 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.057291 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.057273 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.057328 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057297 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.057328 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057312 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.057521 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.057504 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.057558 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057525 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.057558 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057540 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.057723 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.057707 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.057788 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057730 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.057788 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057749 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.057953 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.057934 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.058023 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057957 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.058023 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.057969 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.058171 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:49:28.058156 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.058213 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058176 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.058213 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058192 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.058377 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058359 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.058414 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058378 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.058556 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058540 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.058622 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058559 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.058793 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058773 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.058793 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058791 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.058997 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058960 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.058997 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.058992 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.059192 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059175 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.059239 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059193 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.059381 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059366 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.059424 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059381 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.059580 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059563 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.059621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059582 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.059810 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059794 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.059857 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059810 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.060052 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.059968 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.060105 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060053 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.060259 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060242 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.060309 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060260 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.060480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060458 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.060480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060480 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.060678 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060663 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.060725 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060677 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.060848 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060833 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.060904 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.060847 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.061077 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061061 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.061129 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061077 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.061265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061250 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.061306 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061264 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.061435 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061421 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.061477 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061434 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.061617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061598 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.061679 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061618 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.061837 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061818 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.061837 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.061836 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.062039 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062020 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.062084 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062041 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.062238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062221 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.062278 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062239 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.062412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062399 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.062461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062412 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.062577 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062561 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.062624 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062578 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.062755 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062738 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.062795 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062754 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.062918 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062903 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.062958 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.062919 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.063168 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063150 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.063213 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063168 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.063372 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063355 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.063420 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063373 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.063589 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063570 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.063644 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063590 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.063829 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063813 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.063878 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.063830 2574 scope.go:117] "RemoveContainer" containerID="43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490" Jul 29 15:49:28.064057 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064034 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490"} err="failed to get container status \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": rpc error: code = NotFound desc = could not find container \"43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490\": container with ID starting with 43a502f6a3118c415a5ee5c2713ff06ca5609a7870edd69522e40e022feaf490 not found: ID does not exist" Jul 29 15:49:28.064107 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064059 2574 scope.go:117] "RemoveContainer" containerID="524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8" Jul 29 15:49:28.064468 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064442 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8"} err="failed to get container status \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": rpc error: code = NotFound desc = could not find container \"524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8\": container with ID starting with 524e3d27a368345efdee4d5de1ac5d7a543e86a68d1ef2c34976bd922bcf90c8 not found: ID does not exist" Jul 29 15:49:28.064585 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064470 2574 scope.go:117] "RemoveContainer" containerID="e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f" Jul 29 15:49:28.064804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064778 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f"} err="failed to get container status \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": rpc error: code = NotFound desc = could not find container \"e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f\": container with ID starting with e2b32fa1467823cd2964e4d2c3e14d1fea0fe94a08281ffdd6591fac3e26967f not found: ID does not exist" Jul 29 15:49:28.064804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.064803 2574 scope.go:117] "RemoveContainer" containerID="a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa" Jul 29 15:49:28.065093 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065056 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa"} err="failed to get container status \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": rpc error: code = NotFound desc = could not find container \"a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa\": container with ID starting with a15bd9aa3a801123d19b5cfea6887865d222e9b3eae54d4934dbcedc886bb3fa not found: ID does not exist" Jul 29 15:49:28.065093 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065089 2574 scope.go:117] "RemoveContainer" containerID="d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6" Jul 29 15:49:28.065366 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065344 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6"} err="failed to get container status \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": rpc error: code = NotFound desc = could not find container \"d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6\": container with ID starting with d6d67da2da0569aff71705926b29159432c414f56969e488e8b0f86653ff93f6 not found: ID does not exist" Jul 29 15:49:28.065425 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065366 2574 scope.go:117] "RemoveContainer" containerID="0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b" Jul 29 15:49:28.065632 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065582 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b"} err="failed to get container status \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": rpc error: code = NotFound desc = could not find container \"0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b\": container with ID starting with 0e01b61f33c9054044093589260bfb361f3658b10c5c007fe2c3405ad12b825b not found: ID does not exist" Jul 29 15:49:28.065680 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065633 2574 scope.go:117] "RemoveContainer" containerID="0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c" Jul 29 15:49:28.065900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.065879 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c"} err="failed to get container status \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": rpc error: code = NotFound desc = could not find container \"0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c\": container with ID starting with 0dc6fc7196b965ed65ce8f4bcb0c886aa2ef0820193517783f501f93e48e409c not found: ID does not exist" Jul 29 15:49:28.066126 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066109 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:28.066498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066480 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-thanos" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066500 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-thanos" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066517 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="init-config-reloader" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066527 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="init-config-reloader" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066539 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="thanos-sidecar" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066548 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="thanos-sidecar" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066566 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy" Jul 29 15:49:28.066582 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066581 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="prometheus" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066590 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="prometheus" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066602 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="config-reloader" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="config-reloader" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066647 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-web" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066656 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-web" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066722 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="prometheus" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066732 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-thanos" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066742 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy-web" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066752 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="thanos-sidecar" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066760 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="kube-rbac-proxy" Jul 29 15:49:28.066953 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.066771 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" containerName="config-reloader" Jul 29 15:49:28.070854 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.070835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.073801 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073785 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bupb5r3gdhidh\"" Jul 29 15:49:28.073893 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073840 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 15:49:28.073893 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073871 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 15:49:28.073893 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073884 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 15:49:28.074090 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073892 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 15:49:28.074090 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.073921 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fnwgf\"" Jul 29 15:49:28.074190 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074142 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 15:49:28.074242 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074204 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 15:49:28.074294 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074252 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 15:49:28.074421 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 15:49:28.074471 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074423 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 15:49:28.074471 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074416 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 15:49:28.074540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074502 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 15:49:28.074540 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074510 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 15:49:28.074642 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.074627 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 15:49:28.078128 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.078102 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 15:49:28.080436 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.080416 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 15:49:28.087842 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.087814 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:28.089488 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089467 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089722 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnnw\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-kube-api-access-lpnnw\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.089928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.089998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.090265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.090181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.190621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.190621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.190621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.190621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.190621 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnnw\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-kube-api-access-lpnnw\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.190970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191092 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.191865 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.191634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.192110 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.192086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.192254 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.192132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.192447 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.192420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.192660 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.192632 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.193466 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.193438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.193919 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.193894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.194238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.194214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.194934 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.194593 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.194934 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.194893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.195550 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.195495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-config\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.195944 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.195916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196064 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.195999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196202 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.196184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196344 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.196324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196487 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.196469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec0e087f-1f24-472c-8373-1fabcd53b937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196687 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.196670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec0e087f-1f24-472c-8373-1fabcd53b937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.196774 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.196758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec0e087f-1f24-472c-8373-1fabcd53b937-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.199395 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.199378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnnw\" (UniqueName: \"kubernetes.io/projected/ec0e087f-1f24-472c-8373-1fabcd53b937-kube-api-access-lpnnw\") pod \"prometheus-k8s-0\" (UID: \"ec0e087f-1f24-472c-8373-1fabcd53b937\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.332360 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.332329 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf75ab9-cc9b-4334-bce2-a11f01e7e941" path="/var/lib/kubelet/pods/5bf75ab9-cc9b-4334-bce2-a11f01e7e941/volumes" Jul 29 15:49:28.381189 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.381142 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:28.521907 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:28.521876 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 15:49:28.524895 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:49:28.524858 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0e087f_1f24_472c_8373_1fabcd53b937.slice/crio-8f051e741fcfc206e110f74b91d3a1f4a20bf26659cf35c40fe3810989b497a2 WatchSource:0}: Error finding container 8f051e741fcfc206e110f74b91d3a1f4a20bf26659cf35c40fe3810989b497a2: Status 404 returned error can't find the container with id 8f051e741fcfc206e110f74b91d3a1f4a20bf26659cf35c40fe3810989b497a2 Jul 29 15:49:29.017035 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:29.016998 2574 generic.go:356] "Generic (PLEG): container finished" podID="ec0e087f-1f24-472c-8373-1fabcd53b937" containerID="4e26e02cb2142c5092709eac4c643076686bd082a726ad96f6e36aa848435ac1" exitCode=0 Jul 29 15:49:29.017484 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:29.017069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerDied","Data":"4e26e02cb2142c5092709eac4c643076686bd082a726ad96f6e36aa848435ac1"} Jul 29 15:49:29.017484 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:29.017157 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"8f051e741fcfc206e110f74b91d3a1f4a20bf26659cf35c40fe3810989b497a2"} Jul 29 15:49:30.027558 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"f43bf8e6f40bdfd9f1fc4c078f660ae9adf37aad335799a6e21d99599d21af97"} Jul 29 15:49:30.027947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"dbb097751d6c798376937c7e122c00cea673b87e37a0d570be9663477362c9f8"} Jul 29 15:49:30.027947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"eafc1080375defc6750429bd276540a73100a8b08e135a05d375ea3f9ca9b2c9"} Jul 29 15:49:30.027947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"7ed02946612863ecb9635538e8c67b9ccc39e1d1e52da6d55d480eb5cf629c0f"} Jul 29 15:49:30.027947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"668eb3b8717a34d2d0d4c396a619b43be5ab8153509a388cf6f15d0408a5b2ca"} Jul 29 15:49:30.027947 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.027699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec0e087f-1f24-472c-8373-1fabcd53b937","Type":"ContainerStarted","Data":"f6cc03513c2afb898a10edd9280a5ade90a2eef84d3e4f0879c12d125889368c"} Jul 29 15:49:30.061107 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.061046 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.042576468 podStartE2EDuration="2.061026381s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.449913ms" isStatefulPod=false podCreationTimestamp="2026-07-29 15:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:49:30.059293556 +0000 UTC m=+166.297850784" watchObservedRunningTime="2026-07-29 15:49:30.061026381 +0000 UTC m=+166.299583591" Jul 29 15:49:30.391480 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.391400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:30.454171 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:30.454142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:31.045487 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:31.045456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:31.146040 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:31.145995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:31.236620 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:31.236587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:33.381505 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:33.381452 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:49:35.610601 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:35.610496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:35.712037 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:35.711999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:54.605312 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:54.605268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:49:54.691038 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:49:54.691005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:26.561763 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:26.561726 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:26.661930 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:26.661896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:28.382050 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:28.382003 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:50:28.397332 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:28.397298 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:50:29.250287 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:29.250258 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 15:50:46.302987 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:46.302941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:48.445608 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:48.445578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:49.243578 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:49.243537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:52.050107 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:52.050074 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:52.151239 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:52.151206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:54.133991 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:54.133945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:54.232192 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:54.232159 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:54.585253 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:54.585222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:54.742261 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:54.742222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:56.103007 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:56.102957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:56.484248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:56.484217 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:59.051706 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:59.051670 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:50:59.112931 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:50:59.112902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:05.205393 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:05.205364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:05.308590 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:05.308560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:05.406339 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:05.406311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:07.971227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:07.971192 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:08.037132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:08.037093 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:12.887607 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:12.887574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:12.988916 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:12.988885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:32.490688 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:32.490651 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:32.594878 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:32.594850 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:38.232862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:38.232828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:38.339268 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:38.339237 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:40.354588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:40.354553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:44.174756 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:44.174728 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:51:44.182527 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:44.182504 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 15:51:44.184679 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:44.184659 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:51:44.633761 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:44.633722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:44.719468 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:44.719438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:52.041933 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:52.041902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:53.845808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:53.845776 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:53.946119 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:53.946086 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:54.501783 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.501749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:54.813721 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.813642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:54.921618 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.921586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:54.947031 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.947003 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn"] Jul 29 15:51:54.949173 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.949159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:54.952334 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.952317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 15:51:54.952928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.952901 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k69gf\"" Jul 29 15:51:54.953008 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.952933 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 15:51:54.953209 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.953194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 15:51:54.955744 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.955720 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 15:51:54.969728 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:54.969702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn"] Jul 29 15:51:55.074137 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.074054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.074137 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.074102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.074327 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.074189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.074327 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.074231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.074327 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.074262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwcv6\" (UniqueName: \"kubernetes.io/projected/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-kube-api-access-rwcv6\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.174754 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.174721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwcv6\" (UniqueName: \"kubernetes.io/projected/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-kube-api-access-rwcv6\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.174754 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.174767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.174974 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.174795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.174974 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.174848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.174974 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.174879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.175366 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.175343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-manifests\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.177147 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.177127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-tmp\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.177252 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.177233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.177350 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.177334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.189954 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.189928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwcv6\" (UniqueName: \"kubernetes.io/projected/ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0-kube-api-access-rwcv6\") pod \"opendatahub-operator-controller-manager-75b794959f-7dsrn\" (UID: \"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0\") " pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.232619 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.232583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.259608 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.259578 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:51:55.332462 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.332395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.406178 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.406157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn"] Jul 29 15:51:55.408298 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:51:55.408273 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3e5bee_1e34_46da_b5e6_23a9c3fb34d0.slice/crio-e1ff1a9c5e87d5a9bd6d3dcbe1e3d97c4979330ad1d7318d740792782ae365e1 WatchSource:0}: Error finding container e1ff1a9c5e87d5a9bd6d3dcbe1e3d97c4979330ad1d7318d740792782ae365e1: Status 404 returned error can't find the container with id e1ff1a9c5e87d5a9bd6d3dcbe1e3d97c4979330ad1d7318d740792782ae365e1 Jul 29 15:51:55.410087 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.410071 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:51:55.429814 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.429787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.526811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.526771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" event={"ID":"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0","Type":"ContainerStarted","Data":"e1ff1a9c5e87d5a9bd6d3dcbe1e3d97c4979330ad1d7318d740792782ae365e1"} Jul 29 15:51:55.527138 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.527118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.630551 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.630488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.730268 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.730239 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:55.961604 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:55.961573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:56.062174 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:56.062145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:58.034490 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:58.034453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:58.135849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:58.135817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:51:58.539376 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:58.539345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" event={"ID":"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0","Type":"ContainerStarted","Data":"ed610aefb8aef2871e317dc7185065538e0c4399ee3484b0ec99c7d2fdec88a2"} Jul 29 15:51:59.544278 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:59.544250 2574 generic.go:356] "Generic (PLEG): container finished" podID="ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0" containerID="ed610aefb8aef2871e317dc7185065538e0c4399ee3484b0ec99c7d2fdec88a2" exitCode=0 Jul 29 15:51:59.544589 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:51:59.544336 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" event={"ID":"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0","Type":"ContainerDied","Data":"ed610aefb8aef2871e317dc7185065538e0c4399ee3484b0ec99c7d2fdec88a2"} Jul 29 15:52:00.549719 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:00.549684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" event={"ID":"ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0","Type":"ContainerStarted","Data":"150cef6231c6311ace32850c550c67a2aef6452220bb06827648b151be0ae329"} Jul 29 15:52:00.550128 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:00.549819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:52:00.573646 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:00.573599 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" podStartSLOduration=2.193661631 podStartE2EDuration="6.573584191s" totalImagesPullingTime="3.368001723s" totalInitContainerRuntime="1.011920837s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:51:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:00.571422755 +0000 UTC m=+316.809979966" watchObservedRunningTime="2026-07-29 15:52:00.573584191 +0000 UTC m=+316.812141400" Jul 29 15:52:06.401120 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:06.401088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:06.495613 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:06.495584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:11.555996 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:11.555947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75b794959f-7dsrn" Jul 29 15:52:14.044805 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.044772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:14.253546 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.253511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:14.797803 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.797768 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7cwgg"] Jul 29 15:52:14.804944 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.804919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:14.807617 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.807587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 15:52:14.807728 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.807588 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x7pj6\"" Jul 29 15:52:14.814226 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.814206 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7cwgg"] Jul 29 15:52:14.830281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.830250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:14.830404 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.830286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gw7j\" (UniqueName: \"kubernetes.io/projected/1d38911e-b47a-4819-97b5-7c8d37cc113c-kube-api-access-9gw7j\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:14.840809 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.840761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:14.906258 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.906226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:14.931662 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.931628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:14.931806 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.931673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gw7j\" (UniqueName: \"kubernetes.io/projected/1d38911e-b47a-4819-97b5-7c8d37cc113c-kube-api-access-9gw7j\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:14.931861 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:14.931797 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 15:52:14.931897 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:14.931872 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert podName:1d38911e-b47a-4819-97b5-7c8d37cc113c nodeName:}" failed. No retries permitted until 2026-07-29 15:52:15.431850251 +0000 UTC m=+331.670407454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert") pod "odh-model-controller-7998c67dc4-7cwgg" (UID: "1d38911e-b47a-4819-97b5-7c8d37cc113c") : secret "odh-model-controller-webhook-cert" not found Jul 29 15:52:14.945537 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:14.945503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gw7j\" (UniqueName: \"kubernetes.io/projected/1d38911e-b47a-4819-97b5-7c8d37cc113c-kube-api-access-9gw7j\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:15.013228 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.013187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:15.330499 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.330464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:15.436364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.436322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:15.438694 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.438675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d38911e-b47a-4819-97b5-7c8d37cc113c-cert\") pod \"odh-model-controller-7998c67dc4-7cwgg\" (UID: \"1d38911e-b47a-4819-97b5-7c8d37cc113c\") " pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:15.553547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.553507 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p4fbj"] Jul 29 15:52:15.555952 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.555936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.559030 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.559008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 15:52:15.598009 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.597937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:15.638720 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.638689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.638839 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.638737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngj2w\" (UniqueName: \"kubernetes.io/projected/565bce0f-3411-4174-b34f-421d8f434a61-kube-api-access-ngj2w\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.696890 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.696867 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p4fbj"] Jul 29 15:52:15.716298 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.716274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:15.739713 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.739686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.739877 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.739732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj2w\" (UniqueName: \"kubernetes.io/projected/565bce0f-3411-4174-b34f-421d8f434a61-kube-api-access-ngj2w\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.739877 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:15.739864 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 29 15:52:15.740044 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:15.739945 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs podName:565bce0f-3411-4174-b34f-421d8f434a61 nodeName:}" failed. No retries permitted until 2026-07-29 15:52:16.239924483 +0000 UTC m=+332.478481688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs") pod "model-serving-api-54d89bd5fc-p4fbj" (UID: "565bce0f-3411-4174-b34f-421d8f434a61") : secret "model-serving-api-tls" not found Jul 29 15:52:15.760011 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.759958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj2w\" (UniqueName: \"kubernetes.io/projected/565bce0f-3411-4174-b34f-421d8f434a61-kube-api-access-ngj2w\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:15.779560 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.779526 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2"] Jul 29 15:52:15.782931 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.782900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.786185 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.786139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 15:52:15.787192 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.786229 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 15:52:15.787192 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.786419 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 15:52:15.787192 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.786635 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 15:52:15.788625 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.788599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:15.795238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.795218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2"] Jul 29 15:52:15.841588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.841550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tls-certs\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.841772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.841612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tmp\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.841772 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.841729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qkmr\" (UniqueName: \"kubernetes.io/projected/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-kube-api-access-4qkmr\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.860696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.860617 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7cwgg"] Jul 29 15:52:15.864305 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:15.864273 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d38911e_b47a_4819_97b5_7c8d37cc113c.slice/crio-5e5c4bc479832643aab851be675a7e324868f4c7c9811a9dc8738600c4c6f702 WatchSource:0}: Error finding container 5e5c4bc479832643aab851be675a7e324868f4c7c9811a9dc8738600c4c6f702: Status 404 returned error can't find the container with id 5e5c4bc479832643aab851be675a7e324868f4c7c9811a9dc8738600c4c6f702 Jul 29 15:52:15.881668 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.881647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:15.943054 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.943017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tmp\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.943250 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.943086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qkmr\" (UniqueName: \"kubernetes.io/projected/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-kube-api-access-4qkmr\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.943250 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.943138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tls-certs\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.945385 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.945362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tmp\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.945635 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.945613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-tls-certs\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:15.952364 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:15.952341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qkmr\" (UniqueName: \"kubernetes.io/projected/84385d44-1b71-4ada-b28c-a24c8dc9a9bb-kube-api-access-4qkmr\") pod \"kube-auth-proxy-86bbb74d46-6q8p2\" (UID: \"84385d44-1b71-4ada-b28c-a24c8dc9a9bb\") " pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:16.095351 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.095313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" Jul 29 15:52:16.244073 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.244039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2"] Jul 29 15:52:16.246812 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.246783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:16.247872 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:16.247844 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84385d44_1b71_4ada_b28c_a24c8dc9a9bb.slice/crio-441856b9a70dbec6da7055dd306d6a8a739e738753b78583add78c01ebf46598 WatchSource:0}: Error finding container 441856b9a70dbec6da7055dd306d6a8a739e738753b78583add78c01ebf46598: Status 404 returned error can't find the container with id 441856b9a70dbec6da7055dd306d6a8a739e738753b78583add78c01ebf46598 Jul 29 15:52:16.249294 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.249270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565bce0f-3411-4174-b34f-421d8f434a61-tls-certs\") pod \"model-serving-api-54d89bd5fc-p4fbj\" (UID: \"565bce0f-3411-4174-b34f-421d8f434a61\") " pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:16.466295 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.466263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:16.606651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.606612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" event={"ID":"1d38911e-b47a-4819-97b5-7c8d37cc113c","Type":"ContainerStarted","Data":"5e5c4bc479832643aab851be675a7e324868f4c7c9811a9dc8738600c4c6f702"} Jul 29 15:52:16.607749 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.607718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" event={"ID":"84385d44-1b71-4ada-b28c-a24c8dc9a9bb","Type":"ContainerStarted","Data":"441856b9a70dbec6da7055dd306d6a8a739e738753b78583add78c01ebf46598"} Jul 29 15:52:16.639926 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:16.639880 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p4fbj"] Jul 29 15:52:16.642478 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:16.642444 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565bce0f_3411_4174_b34f_421d8f434a61.slice/crio-296b7d178e43e4c0a0ed2aafa493e2619ffa0d9b0d3916b32fd7093fadfdf33c WatchSource:0}: Error finding container 296b7d178e43e4c0a0ed2aafa493e2619ffa0d9b0d3916b32fd7093fadfdf33c: Status 404 returned error can't find the container with id 296b7d178e43e4c0a0ed2aafa493e2619ffa0d9b0d3916b32fd7093fadfdf33c Jul 29 15:52:17.549187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.547633 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:17.549187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.548597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:17.550579 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.550551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.554994 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.554963 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-clx7j\"" Jul 29 15:52:17.555201 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.555185 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 15:52:17.556443 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.556425 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 15:52:17.556645 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.556625 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 15:52:17.556813 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.556788 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 15:52:17.569725 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.569701 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:17.620850 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.620811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" event={"ID":"565bce0f-3411-4174-b34f-421d8f434a61","Type":"ContainerStarted","Data":"296b7d178e43e4c0a0ed2aafa493e2619ffa0d9b0d3916b32fd7093fadfdf33c"} Jul 29 15:52:17.663508 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663508 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663508 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663508 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9f6\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663508 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663896 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.663896 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.663565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.670769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.670702 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:17.764099 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.764498 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.764308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9f6\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.765881 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.765531 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.768545 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.768498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.768991 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.768959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.779630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.771721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.779630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.774317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.779630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.777896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9f6\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.779630 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.778322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wxgl7\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:17.868586 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:17.868504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:18.960631 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:18.960595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:20.274030 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.274005 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:20.276619 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:20.276593 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee25802b_f77d_4c77_85fe_79d0292533ab.slice/crio-170c715bd6188350321f312ddd04d4ddbed13fed6d9da5b9b224f518a9b3153a WatchSource:0}: Error finding container 170c715bd6188350321f312ddd04d4ddbed13fed6d9da5b9b224f518a9b3153a: Status 404 returned error can't find the container with id 170c715bd6188350321f312ddd04d4ddbed13fed6d9da5b9b224f518a9b3153a Jul 29 15:52:20.637951 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.637911 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" event={"ID":"565bce0f-3411-4174-b34f-421d8f434a61","Type":"ContainerStarted","Data":"831307c22c1b05b5452ad408d1dc2200647f8f5f3207762239c6ddb4530a6b34"} Jul 29 15:52:20.638178 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.638031 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:20.639338 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.639312 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" event={"ID":"ee25802b-f77d-4c77-85fe-79d0292533ab","Type":"ContainerStarted","Data":"170c715bd6188350321f312ddd04d4ddbed13fed6d9da5b9b224f518a9b3153a"} Jul 29 15:52:20.640863 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.640836 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" event={"ID":"1d38911e-b47a-4819-97b5-7c8d37cc113c","Type":"ContainerStarted","Data":"34cf62f3f0d1f943805cbc47fbd055e2eaf3dd36436808095c5e37def03f9deb"} Jul 29 15:52:20.640990 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.640947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:20.642399 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.642367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" event={"ID":"84385d44-1b71-4ada-b28c-a24c8dc9a9bb","Type":"ContainerStarted","Data":"10ca227ffcaaa366ff71e9212784ad592b188db33350f97e46d7b021af98e3d0"} Jul 29 15:52:20.658863 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.658821 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" podStartSLOduration=2.183456567 podStartE2EDuration="5.658808166s" totalImagesPullingTime="3.475351599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:20.657892121 +0000 UTC m=+336.896449343" watchObservedRunningTime="2026-07-29 15:52:20.658808166 +0000 UTC m=+336.897365403" Jul 29 15:52:20.676506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.676464 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86bbb74d46-6q8p2" podStartSLOduration=1.806703707 podStartE2EDuration="5.676452088s" totalImagesPullingTime="3.869748381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:20.675568274 +0000 UTC m=+336.914125484" watchObservedRunningTime="2026-07-29 15:52:20.676452088 +0000 UTC m=+336.915009297" Jul 29 15:52:20.697796 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:20.697745 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" podStartSLOduration=2.44786855 podStartE2EDuration="6.697730323s" totalImagesPullingTime="4.249861773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:20.695843064 +0000 UTC m=+336.934400274" watchObservedRunningTime="2026-07-29 15:52:20.697730323 +0000 UTC m=+336.936287532" Jul 29 15:52:23.051158 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.051120 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:52:23.051419 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.051190 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:52:23.659171 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.659132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" event={"ID":"ee25802b-f77d-4c77-85fe-79d0292533ab","Type":"ContainerStarted","Data":"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da"} Jul 29 15:52:23.659349 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.659231 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:23.709180 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.709123 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" podStartSLOduration=3.936854334 podStartE2EDuration="6.709106883s" totalImagesPullingTime="2.772252549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:23.707872603 +0000 UTC m=+339.946429814" watchObservedRunningTime="2026-07-29 15:52:23.709106883 +0000 UTC m=+339.947664092" Jul 29 15:52:23.926668 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:23.926589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:24.100502 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.100466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:24.486387 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.486357 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h"] Jul 29 15:52:24.489849 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.489832 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.522743 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.522718 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dmnqs\"" Jul 29 15:52:24.526997 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.526952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254bd7-29dd-40a3-8f7d-408516ea47cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrxp\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-kube-api-access-2vrxp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527287 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527287 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.527287 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.527250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.564740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.564713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:24.572876 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.572852 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h"] Jul 29 15:52:24.628115 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrxp\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-kube-api-access-2vrxp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628251 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628251 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628251 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254bd7-29dd-40a3-8f7d-408516ea47cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628412 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628704 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.628942 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.629057 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.628946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.629057 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.629024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.629795 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.629776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254bd7-29dd-40a3-8f7d-408516ea47cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.630999 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.630958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.630999 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.630963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.645502 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.645478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.654143 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.654123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrxp\" (UniqueName: \"kubernetes.io/projected/38254bd7-29dd-40a3-8f7d-408516ea47cf-kube-api-access-2vrxp\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h\" (UID: \"38254bd7-29dd-40a3-8f7d-408516ea47cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.664966 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.664947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:24.800218 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.800133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:24.854324 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:24.854282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:25.003952 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:25.003549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h"] Jul 29 15:52:25.120252 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:25.120181 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:25.668055 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:25.668014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" event={"ID":"38254bd7-29dd-40a3-8f7d-408516ea47cf","Type":"ContainerStarted","Data":"6b7d39aacaf2f054417e7d8183fdc8a615becda11fcda6f97d4a982f6a5050d3"} Jul 29 15:52:28.349798 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.349751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:28.526692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.526659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:28.537773 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.537745 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb"] Jul 29 15:52:28.544442 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.544421 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn"] Jul 29 15:52:28.544575 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.544490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.547708 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.547689 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.557292 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.557272 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 15:52:28.557428 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.557278 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z4lz2\"" Jul 29 15:52:28.557788 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.557636 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2nw9r\"" Jul 29 15:52:28.557788 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.557661 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 15:52:28.585871 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.585841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb"] Jul 29 15:52:28.587869 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.587843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn"] Jul 29 15:52:28.667185 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.667119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldcg\" (UniqueName: \"kubernetes.io/projected/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-kube-api-access-cldcg\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.667185 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.667162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.667375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.667235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lds4\" (UniqueName: \"kubernetes.io/projected/c84d81c9-379b-4dc5-899e-6b2d027625bf-kube-api-access-9lds4\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.667375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.667282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84d81c9-379b-4dc5-899e-6b2d027625bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.728021 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.727970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:28.728312 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.728292 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:28.728523 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.728496 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" podUID="ee25802b-f77d-4c77-85fe-79d0292533ab" containerName="discovery" containerID="cri-o://58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da" gracePeriod=30 Jul 29 15:52:28.768169 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.768143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.768271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.768230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lds4\" (UniqueName: \"kubernetes.io/projected/c84d81c9-379b-4dc5-899e-6b2d027625bf-kube-api-access-9lds4\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.768339 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:28.768290 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 29 15:52:28.768339 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.768295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84d81c9-379b-4dc5-899e-6b2d027625bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.768428 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:28.768357 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert podName:6e74ea06-8bbb-42d5-abd4-94d21eae1b44 nodeName:}" failed. No retries permitted until 2026-07-29 15:52:29.268336695 +0000 UTC m=+345.506893899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert") pod "kserve-controller-manager-d6f4b65b8-x7jdn" (UID: "6e74ea06-8bbb-42d5-abd4-94d21eae1b44") : secret "kserve-webhook-server-cert" not found Jul 29 15:52:28.768428 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.768378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldcg\" (UniqueName: \"kubernetes.io/projected/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-kube-api-access-cldcg\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.770928 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.770907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84d81c9-379b-4dc5-899e-6b2d027625bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.781700 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.781675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lds4\" (UniqueName: \"kubernetes.io/projected/c84d81c9-379b-4dc5-899e-6b2d027625bf-kube-api-access-9lds4\") pod \"llmisvc-controller-manager-777c6bd7d5-8mmvb\" (UID: \"c84d81c9-379b-4dc5-899e-6b2d027625bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:28.785181 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.785163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldcg\" (UniqueName: \"kubernetes.io/projected/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-kube-api-access-cldcg\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:28.847197 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.847165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:28.858039 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:28.858014 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:29.000308 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.000275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:52:29.280046 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.279947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:29.282653 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.282623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e74ea06-8bbb-42d5-abd4-94d21eae1b44-cert\") pod \"kserve-controller-manager-d6f4b65b8-x7jdn\" (UID: \"6e74ea06-8bbb-42d5-abd4-94d21eae1b44\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:29.464992 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.464950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:29.495510 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.495471 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:52:29.495615 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.495565 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:52:29.495615 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.495605 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:52:29.588516 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.588484 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:29.610428 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.610315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb"] Jul 29 15:52:29.616678 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:29.616651 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc84d81c9_379b_4dc5_899e_6b2d027625bf.slice/crio-852ab85b3393c500d2d2921704fca28bc86af97e24ebacb49b36f7fe5664bcf7 WatchSource:0}: Error finding container 852ab85b3393c500d2d2921704fca28bc86af97e24ebacb49b36f7fe5664bcf7: Status 404 returned error can't find the container with id 852ab85b3393c500d2d2921704fca28bc86af97e24ebacb49b36f7fe5664bcf7 Jul 29 15:52:29.684372 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684342 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.684547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684384 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.684547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.684547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x9f6\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.684547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684505 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.684547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn"] Jul 29 15:52:29.685023 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684568 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.685023 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.684626 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap\") pod \"ee25802b-f77d-4c77-85fe-79d0292533ab\" (UID: \"ee25802b-f77d-4c77-85fe-79d0292533ab\") " Jul 29 15:52:29.686892 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.686833 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 15:52:29.688168 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.688137 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:52:29.688256 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.688177 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs" (OuterVolumeSpecName: "local-certs") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:52:29.688256 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.688241 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:52:29.688358 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.688242 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6" (OuterVolumeSpecName: "kube-api-access-7x9f6") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "kube-api-access-7x9f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:52:29.689195 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.689138 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:52:29.689654 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.689632 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts" (OuterVolumeSpecName: "cacerts") pod "ee25802b-f77d-4c77-85fe-79d0292533ab" (UID: "ee25802b-f77d-4c77-85fe-79d0292533ab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:52:29.690206 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.690181 2574 generic.go:356] "Generic (PLEG): container finished" podID="ee25802b-f77d-4c77-85fe-79d0292533ab" containerID="58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da" exitCode=0 Jul 29 15:52:29.690304 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.690232 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" Jul 29 15:52:29.690304 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.690260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" event={"ID":"ee25802b-f77d-4c77-85fe-79d0292533ab","Type":"ContainerDied","Data":"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da"} Jul 29 15:52:29.690444 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.690316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7" event={"ID":"ee25802b-f77d-4c77-85fe-79d0292533ab","Type":"ContainerDied","Data":"170c715bd6188350321f312ddd04d4ddbed13fed6d9da5b9b224f518a9b3153a"} Jul 29 15:52:29.690444 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.690342 2574 scope.go:117] "RemoveContainer" containerID="58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da" Jul 29 15:52:29.691108 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:52:29.691085 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e74ea06_8bbb_42d5_abd4_94d21eae1b44.slice/crio-ce222e2ab808639db2c958073edf1b68487361bb56059ec50f674648ab029c46 WatchSource:0}: Error finding container ce222e2ab808639db2c958073edf1b68487361bb56059ec50f674648ab029c46: Status 404 returned error can't find the container with id ce222e2ab808639db2c958073edf1b68487361bb56059ec50f674648ab029c46 Jul 29 15:52:29.692694 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.692652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" event={"ID":"38254bd7-29dd-40a3-8f7d-408516ea47cf","Type":"ContainerStarted","Data":"495e240a1a774e16176b8897271028fa55dcad18b8c455811499de68001da8f2"} Jul 29 15:52:29.694563 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.694448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" event={"ID":"c84d81c9-379b-4dc5-899e-6b2d027625bf","Type":"ContainerStarted","Data":"852ab85b3393c500d2d2921704fca28bc86af97e24ebacb49b36f7fe5664bcf7"} Jul 29 15:52:29.700668 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.700652 2574 scope.go:117] "RemoveContainer" containerID="58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da" Jul 29 15:52:29.700913 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:52:29.700888 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da\": container with ID starting with 58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da not found: ID does not exist" containerID="58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da" Jul 29 15:52:29.701018 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.700928 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da"} err="failed to get container status \"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da\": rpc error: code = NotFound desc = could not find container \"58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da\": container with ID starting with 58446c5868d667fe9fba0931e476f285aa15289fa1e011221de3bd0c743f85da not found: ID does not exist" Jul 29 15:52:29.727544 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.727483 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" podStartSLOduration=1.242398592 podStartE2EDuration="5.72746488s" totalImagesPullingTime="4.485066288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:29.725331975 +0000 UTC m=+345.963889184" watchObservedRunningTime="2026-07-29 15:52:29.72746488 +0000 UTC m=+345.966022091" Jul 29 15:52:29.743816 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.743790 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:29.746625 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.746604 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wxgl7"] Jul 29 15:52:29.786379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786305 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee25802b-f77d-4c77-85fe-79d0292533ab-local-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786333 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-dns-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786349 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-kubeconfig\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786364 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9f6\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-kube-api-access-7x9f6\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786379 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786379 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786652 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786391 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee25802b-f77d-4c77-85fe-79d0292533ab-cacerts\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.786652 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.786404 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee25802b-f77d-4c77-85fe-79d0292533ab-istio-csr-ca-configmap\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:52:29.801061 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:29.801039 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:30.334424 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:30.334394 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee25802b-f77d-4c77-85fe-79d0292533ab" path="/var/lib/kubelet/pods/ee25802b-f77d-4c77-85fe-79d0292533ab/volumes" Jul 29 15:52:30.701505 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:30.701427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" event={"ID":"6e74ea06-8bbb-42d5-abd4-94d21eae1b44","Type":"ContainerStarted","Data":"ce222e2ab808639db2c958073edf1b68487361bb56059ec50f674648ab029c46"} Jul 29 15:52:30.806541 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:30.806510 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:31.651190 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:31.651161 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7cwgg" Jul 29 15:52:31.654095 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:31.653969 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p4fbj" Jul 29 15:52:31.708547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:31.708495 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:31.710058 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:31.709890 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h" Jul 29 15:52:32.713009 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:32.712950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" event={"ID":"6e74ea06-8bbb-42d5-abd4-94d21eae1b44","Type":"ContainerStarted","Data":"60e8176d4d0886066195e2e25fffdc35ef465482475a0fb292ead687ef17fa19"} Jul 29 15:52:32.713299 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:32.713285 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:52:32.732113 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:32.732048 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" podStartSLOduration=2.343984263 podStartE2EDuration="4.732035866s" totalImagesPullingTime="2.388051603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:32.731091347 +0000 UTC m=+348.969648554" watchObservedRunningTime="2026-07-29 15:52:32.732035866 +0000 UTC m=+348.970593075" Jul 29 15:52:33.718231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:33.718192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" event={"ID":"c84d81c9-379b-4dc5-899e-6b2d027625bf","Type":"ContainerStarted","Data":"5778fe3e0bedce8c985e66c96cda6528ae85f5691d027bcb4d82dfe261f9e3d6"} Jul 29 15:52:33.718636 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:33.718269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:52:33.744368 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:52:33.744318 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" podStartSLOduration=2.658909426 podStartE2EDuration="5.744302327s" totalImagesPullingTime="3.085392901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:52:33.742090584 +0000 UTC m=+349.980647798" watchObservedRunningTime="2026-07-29 15:52:33.744302327 +0000 UTC m=+349.982859537" Jul 29 15:53:04.725819 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:04.725792 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8mmvb" Jul 29 15:53:04.728871 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:04.728856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-x7jdn" Jul 29 15:53:15.604066 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.604034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:15.713727 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.713688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:15.730184 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.730155 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:15.730488 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.730474 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee25802b-f77d-4c77-85fe-79d0292533ab" containerName="discovery" Jul 29 15:53:15.730533 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.730491 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee25802b-f77d-4c77-85fe-79d0292533ab" containerName="discovery" Jul 29 15:53:15.730579 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.730549 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee25802b-f77d-4c77-85fe-79d0292533ab" containerName="discovery" Jul 29 15:53:15.732808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.732780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:15.735706 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.735680 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 15:53:15.735833 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.735679 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8rmrp\"" Jul 29 15:53:15.736644 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.736624 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 15:53:15.743060 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.743037 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:15.773161 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.773127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lcc\" (UniqueName: \"kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc\") pod \"kuadrant-operator-catalog-4t6gk\" (UID: \"e0235b93-55ab-4701-acf5-7145bbaada40\") " pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:15.810248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.810214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:15.874572 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.874497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lcc\" (UniqueName: \"kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc\") pod \"kuadrant-operator-catalog-4t6gk\" (UID: \"e0235b93-55ab-4701-acf5-7145bbaada40\") " pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:15.883023 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:15.882973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lcc\" (UniqueName: \"kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc\") pod \"kuadrant-operator-catalog-4t6gk\" (UID: \"e0235b93-55ab-4701-acf5-7145bbaada40\") " pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:16.044587 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.044548 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:16.173661 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.173636 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:16.176173 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:53:16.176140 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0235b93_55ab_4701_acf5_7145bbaada40.slice/crio-50964554454b84927b467ad1ae134180c5c86d4fe4b0745b484cfa10de61fe78 WatchSource:0}: Error finding container 50964554454b84927b467ad1ae134180c5c86d4fe4b0745b484cfa10de61fe78: Status 404 returned error can't find the container with id 50964554454b84927b467ad1ae134180c5c86d4fe4b0745b484cfa10de61fe78 Jul 29 15:53:16.184825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.184800 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:16.295883 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.295851 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-768tz"] Jul 29 15:53:16.299221 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.299206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:16.308247 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.308224 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-768tz"] Jul 29 15:53:16.377854 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.377827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgdh5\" (UniqueName: \"kubernetes.io/projected/6f6e115f-647c-49da-b9f2-6741b2ba8ed4-kube-api-access-xgdh5\") pod \"kuadrant-operator-catalog-768tz\" (UID: \"6f6e115f-647c-49da-b9f2-6741b2ba8ed4\") " pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:16.479036 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.478946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgdh5\" (UniqueName: \"kubernetes.io/projected/6f6e115f-647c-49da-b9f2-6741b2ba8ed4-kube-api-access-xgdh5\") pod \"kuadrant-operator-catalog-768tz\" (UID: \"6f6e115f-647c-49da-b9f2-6741b2ba8ed4\") " pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:16.488574 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.488546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgdh5\" (UniqueName: \"kubernetes.io/projected/6f6e115f-647c-49da-b9f2-6741b2ba8ed4-kube-api-access-xgdh5\") pod \"kuadrant-operator-catalog-768tz\" (UID: \"6f6e115f-647c-49da-b9f2-6741b2ba8ed4\") " pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:16.523640 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.523612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:16.609376 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.609346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:16.726413 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.726377 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:16.739248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.739171 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-768tz"] Jul 29 15:53:16.741365 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:53:16.741334 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e115f_647c_49da_b9f2_6741b2ba8ed4.slice/crio-e43d63c8fda60f62aefeac4fa3f562477b3b32c6087b7836e1b25b046a39bb64 WatchSource:0}: Error finding container e43d63c8fda60f62aefeac4fa3f562477b3b32c6087b7836e1b25b046a39bb64: Status 404 returned error can't find the container with id e43d63c8fda60f62aefeac4fa3f562477b3b32c6087b7836e1b25b046a39bb64 Jul 29 15:53:16.895142 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.895080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" event={"ID":"e0235b93-55ab-4701-acf5-7145bbaada40","Type":"ContainerStarted","Data":"50964554454b84927b467ad1ae134180c5c86d4fe4b0745b484cfa10de61fe78"} Jul 29 15:53:16.896900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:16.896873 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-768tz" event={"ID":"6f6e115f-647c-49da-b9f2-6741b2ba8ed4","Type":"ContainerStarted","Data":"e43d63c8fda60f62aefeac4fa3f562477b3b32c6087b7836e1b25b046a39bb64"} Jul 29 15:53:17.890486 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:17.890450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:18.912614 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:18.912579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-768tz" event={"ID":"6f6e115f-647c-49da-b9f2-6741b2ba8ed4","Type":"ContainerStarted","Data":"25de20f10971922b2b6547ecdf10157b69d0fdeeeae6b44ea9fd355a3bc602d8"} Jul 29 15:53:18.914020 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:18.913994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" event={"ID":"e0235b93-55ab-4701-acf5-7145bbaada40","Type":"ContainerStarted","Data":"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b"} Jul 29 15:53:18.914131 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:18.914058 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" podUID="e0235b93-55ab-4701-acf5-7145bbaada40" containerName="registry-server" containerID="cri-o://de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b" gracePeriod=2 Jul 29 15:53:18.931089 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:18.931046 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-768tz" podStartSLOduration=1.617873993 podStartE2EDuration="2.931030967s" totalImagesPullingTime="1.313156974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:53:18.930573922 +0000 UTC m=+395.169131183" watchObservedRunningTime="2026-07-29 15:53:18.931030967 +0000 UTC m=+395.169588177" Jul 29 15:53:18.950960 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:18.950916 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" podStartSLOduration=2.072883223 podStartE2EDuration="3.950898544s" totalImagesPullingTime="1.878015321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:53:18.94939857 +0000 UTC m=+395.187955780" watchObservedRunningTime="2026-07-29 15:53:18.950898544 +0000 UTC m=+395.189455754" Jul 29 15:53:19.151487 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.151458 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:19.201341 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.201256 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98lcc\" (UniqueName: \"kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc\") pod \"e0235b93-55ab-4701-acf5-7145bbaada40\" (UID: \"e0235b93-55ab-4701-acf5-7145bbaada40\") " Jul 29 15:53:19.203438 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.203401 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc" (OuterVolumeSpecName: "kube-api-access-98lcc") pod "e0235b93-55ab-4701-acf5-7145bbaada40" (UID: "e0235b93-55ab-4701-acf5-7145bbaada40"). InnerVolumeSpecName "kube-api-access-98lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:53:19.302029 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.302000 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lcc\" (UniqueName: \"kubernetes.io/projected/e0235b93-55ab-4701-acf5-7145bbaada40-kube-api-access-98lcc\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:53:19.919132 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.919099 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0235b93-55ab-4701-acf5-7145bbaada40" containerID="de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b" exitCode=0 Jul 29 15:53:19.919568 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.919144 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" Jul 29 15:53:19.919568 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.919192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" event={"ID":"e0235b93-55ab-4701-acf5-7145bbaada40","Type":"ContainerDied","Data":"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b"} Jul 29 15:53:19.919568 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.919243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4t6gk" event={"ID":"e0235b93-55ab-4701-acf5-7145bbaada40","Type":"ContainerDied","Data":"50964554454b84927b467ad1ae134180c5c86d4fe4b0745b484cfa10de61fe78"} Jul 29 15:53:19.919568 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.919268 2574 scope.go:117] "RemoveContainer" containerID="de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b" Jul 29 15:53:19.928072 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.928056 2574 scope.go:117] "RemoveContainer" containerID="de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b" Jul 29 15:53:19.928313 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:53:19.928296 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b\": container with ID starting with de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b not found: ID does not exist" containerID="de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b" Jul 29 15:53:19.928358 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.928323 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b"} err="failed to get container status \"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b\": rpc error: code = NotFound desc = could not find container \"de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b\": container with ID starting with de413d794dc8547f3f7528288ec7c6d452c0e6c01c4cc80bb746737c26739e2b not found: ID does not exist" Jul 29 15:53:19.943225 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.943193 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:19.949307 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:19.949283 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4t6gk"] Jul 29 15:53:20.333223 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:20.333192 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0235b93-55ab-4701-acf5-7145bbaada40" path="/var/lib/kubelet/pods/e0235b93-55ab-4701-acf5-7145bbaada40/volumes" Jul 29 15:53:26.609867 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:26.609822 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:26.610442 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:26.609896 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:26.631704 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:26.631678 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:26.972183 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:26.972153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-768tz" Jul 29 15:53:45.001493 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:45.001420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:47.422346 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.422309 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8"] Jul 29 15:53:47.422736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.422627 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0235b93-55ab-4701-acf5-7145bbaada40" containerName="registry-server" Jul 29 15:53:47.422736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.422637 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0235b93-55ab-4701-acf5-7145bbaada40" containerName="registry-server" Jul 29 15:53:47.422736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.422700 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0235b93-55ab-4701-acf5-7145bbaada40" containerName="registry-server" Jul 29 15:53:47.425660 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.425643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:47.429529 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.429491 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 15:53:47.429529 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.429525 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2rnt7\"" Jul 29 15:53:47.439886 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.439863 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8"] Jul 29 15:53:47.546028 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.545966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxmw\" (UniqueName: \"kubernetes.io/projected/f6a07185-7099-4084-acbb-6dea09eb962a-kube-api-access-qzxmw\") pod \"dns-operator-controller-manager-864dd78c4c-bmzz8\" (UID: \"f6a07185-7099-4084-acbb-6dea09eb962a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:47.647463 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.647421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxmw\" (UniqueName: \"kubernetes.io/projected/f6a07185-7099-4084-acbb-6dea09eb962a-kube-api-access-qzxmw\") pod \"dns-operator-controller-manager-864dd78c4c-bmzz8\" (UID: \"f6a07185-7099-4084-acbb-6dea09eb962a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:47.666937 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.666906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxmw\" (UniqueName: \"kubernetes.io/projected/f6a07185-7099-4084-acbb-6dea09eb962a-kube-api-access-qzxmw\") pod \"dns-operator-controller-manager-864dd78c4c-bmzz8\" (UID: \"f6a07185-7099-4084-acbb-6dea09eb962a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:47.735750 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.735713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:47.866729 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:47.866703 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8"] Jul 29 15:53:47.869023 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:53:47.868995 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a07185_7099_4084_acbb_6dea09eb962a.slice/crio-bafc1b2e671a5dbb211747ca5c02f18f5b1188d748b4f3659a25412c186c2562 WatchSource:0}: Error finding container bafc1b2e671a5dbb211747ca5c02f18f5b1188d748b4f3659a25412c186c2562: Status 404 returned error can't find the container with id bafc1b2e671a5dbb211747ca5c02f18f5b1188d748b4f3659a25412c186c2562 Jul 29 15:53:48.040212 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:48.040128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" event={"ID":"f6a07185-7099-4084-acbb-6dea09eb962a","Type":"ContainerStarted","Data":"bafc1b2e671a5dbb211747ca5c02f18f5b1188d748b4f3659a25412c186c2562"} Jul 29 15:53:48.968116 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:48.968047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:50.054005 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.053952 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" event={"ID":"f6a07185-7099-4084-acbb-6dea09eb962a","Type":"ContainerStarted","Data":"26883fd7294f6c0f164d554b8ff21f63ad2737adab2fef76a0bcd58e70c40145"} Jul 29 15:53:50.054384 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.054038 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:53:50.073198 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.073145 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" podStartSLOduration=1.347906082 podStartE2EDuration="3.07312919s" totalImagesPullingTime="1.725223108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:53:50.072107877 +0000 UTC m=+426.310665102" watchObservedRunningTime="2026-07-29 15:53:50.07312919 +0000 UTC m=+426.311686399" Jul 29 15:53:50.892678 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.892643 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2"] Jul 29 15:53:50.895422 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.895401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:50.898076 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.898056 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nmhjz\"" Jul 29 15:53:50.908004 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:50.906895 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2"] Jul 29 15:53:51.076169 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.076135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jszf\" (UniqueName: \"kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf\") pod \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" (UID: \"189e90a1-c918-41d0-8941-4f64baa24474\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:51.177616 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.177525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jszf\" (UniqueName: \"kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf\") pod \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" (UID: \"189e90a1-c918-41d0-8941-4f64baa24474\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:51.189199 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.189166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jszf\" (UniqueName: \"kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf\") pod \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" (UID: \"189e90a1-c918-41d0-8941-4f64baa24474\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:51.206899 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.206870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:51.339904 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.339878 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2"] Jul 29 15:53:51.342940 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:53:51.342911 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189e90a1_c918_41d0_8941_4f64baa24474.slice/crio-2b6ca6bc60983c48e4302102cba1bf90b7138a0cf12b71d213bcadb7c944939d WatchSource:0}: Error finding container 2b6ca6bc60983c48e4302102cba1bf90b7138a0cf12b71d213bcadb7c944939d: Status 404 returned error can't find the container with id 2b6ca6bc60983c48e4302102cba1bf90b7138a0cf12b71d213bcadb7c944939d Jul 29 15:53:51.370333 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.370274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:51.678750 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.678720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:51.989829 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:51.989796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:52.065325 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:52.065290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" event={"ID":"189e90a1-c918-41d0-8941-4f64baa24474","Type":"ContainerStarted","Data":"2b6ca6bc60983c48e4302102cba1bf90b7138a0cf12b71d213bcadb7c944939d"} Jul 29 15:53:54.075715 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.075682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" event={"ID":"189e90a1-c918-41d0-8941-4f64baa24474","Type":"ContainerStarted","Data":"1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2"} Jul 29 15:53:54.076123 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.075815 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:53:54.097420 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.097369 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" podStartSLOduration=2.425060214 podStartE2EDuration="4.097354066s" totalImagesPullingTime="1.672293852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:53:54.095963219 +0000 UTC m=+430.334520428" watchObservedRunningTime="2026-07-29 15:53:54.097354066 +0000 UTC m=+430.335911276" Jul 29 15:53:54.828932 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.828891 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x"] Jul 29 15:53:54.831248 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.831228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:54.834305 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.834286 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j6dph\"" Jul 29 15:53:54.848280 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:54.848253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x"] Jul 29 15:53:55.011990 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.011950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8t2v\" (UniqueName: \"kubernetes.io/projected/1ba5342d-c3ab-43de-9d78-31369ebc2cf5-kube-api-access-d8t2v\") pod \"authorino-operator-75cc8d5cb7-xdc7x\" (UID: \"1ba5342d-c3ab-43de-9d78-31369ebc2cf5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:55.112534 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.112428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t2v\" (UniqueName: \"kubernetes.io/projected/1ba5342d-c3ab-43de-9d78-31369ebc2cf5-kube-api-access-d8t2v\") pod \"authorino-operator-75cc8d5cb7-xdc7x\" (UID: \"1ba5342d-c3ab-43de-9d78-31369ebc2cf5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:55.126361 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.126327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8t2v\" (UniqueName: \"kubernetes.io/projected/1ba5342d-c3ab-43de-9d78-31369ebc2cf5-kube-api-access-d8t2v\") pod \"authorino-operator-75cc8d5cb7-xdc7x\" (UID: \"1ba5342d-c3ab-43de-9d78-31369ebc2cf5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:55.141259 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.141229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:55.276393 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.276369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x"] Jul 29 15:53:55.379477 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:55.379400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:56.086280 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:56.086245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" event={"ID":"1ba5342d-c3ab-43de-9d78-31369ebc2cf5","Type":"ContainerStarted","Data":"2d4f847ecd332b0e961d78c0895c92ee989994da6bf8f157fe8829fe52dee409"} Jul 29 15:53:56.426451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:56.426370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:56.534577 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:56.534515 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:58.098230 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:58.098143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" event={"ID":"1ba5342d-c3ab-43de-9d78-31369ebc2cf5","Type":"ContainerStarted","Data":"2c137465ce9317bc02db9b679edfdeffa11c34cdba0b4f3428c3abe0d95d5d27"} Jul 29 15:53:58.098588 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:58.098384 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:53:58.120574 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:58.120520 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" podStartSLOduration=1.665414197 podStartE2EDuration="4.120504456s" totalImagesPullingTime="2.455090259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:53:58.118057054 +0000 UTC m=+434.356614265" watchObservedRunningTime="2026-07-29 15:53:58.120504456 +0000 UTC m=+434.359061667" Jul 29 15:53:58.440764 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:58.440688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:53:59.807290 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.807239 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r"] Jul 29 15:53:59.810792 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.810772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.813614 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.813599 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j69w2\"" Jul 29 15:53:59.823341 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.823317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r"] Jul 29 15:53:59.852114 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.852080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mdz\" (UniqueName: \"kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.852286 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.852160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.953461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.953430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mdz\" (UniqueName: \"kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.953635 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.953504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.953870 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.953855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:53:59.961862 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:53:59.961835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mdz\" (UniqueName: \"kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:00.120952 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:00.120858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:00.227201 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:00.227171 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:00.270960 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:00.269341 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r"] Jul 29 15:54:01.060090 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:01.060061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bmzz8" Jul 29 15:54:01.113065 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:01.113036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" event={"ID":"1df01ba8-53f8-4655-a2ba-7cbea823139e","Type":"ContainerStarted","Data":"13f6b7d322a89f4176090375316171eb50f3e6a4fcec0b8f1678185b5e0dd0c4"} Jul 29 15:54:01.640266 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:01.640236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:01.739766 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:01.739718 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:05.082564 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:05.082534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:05.452119 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:05.452033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:05.574934 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:05.574856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:06.137672 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:06.137638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" event={"ID":"1df01ba8-53f8-4655-a2ba-7cbea823139e","Type":"ContainerStarted","Data":"be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec"} Jul 29 15:54:06.138130 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:06.137765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:06.160878 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:06.160821 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" podStartSLOduration=2.201585186 podStartE2EDuration="7.160802733s" totalImagesPullingTime="4.959217547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:06.158768314 +0000 UTC m=+442.397325520" watchObservedRunningTime="2026-07-29 15:54:06.160802733 +0000 UTC m=+442.399359934" Jul 29 15:54:07.917693 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:07.917656 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:08.110209 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:08.110171 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:09.105382 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:09.105345 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xdc7x" Jul 29 15:54:09.635643 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:09.635610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:09.730915 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:09.730882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:09.980119 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:09.980088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:10.083511 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:10.083480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:12.999912 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:12.999881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:13.101100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:13.101069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:17.145678 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:17.145648 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:18.836148 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.836113 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r"] Jul 29 15:54:18.836579 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.836374 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" containerName="manager" containerID="cri-o://be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec" gracePeriod=2 Jul 29 15:54:18.841522 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.841493 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r"] Jul 29 15:54:18.880692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.880659 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2"] Jul 29 15:54:18.880942 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.880903 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" podUID="189e90a1-c918-41d0-8941-4f64baa24474" containerName="manager" containerID="cri-o://1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2" gracePeriod=2 Jul 29 15:54:18.882530 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.882500 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2"] Jul 29 15:54:18.883775 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.883752 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:18.884164 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884146 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189e90a1-c918-41d0-8941-4f64baa24474" containerName="manager" Jul 29 15:54:18.884271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884205 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="189e90a1-c918-41d0-8941-4f64baa24474" containerName="manager" Jul 29 15:54:18.884271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884245 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" containerName="manager" Jul 29 15:54:18.884271 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884254 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" containerName="manager" Jul 29 15:54:18.884420 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.884152 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:18.884420 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884412 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="189e90a1-c918-41d0-8941-4f64baa24474" containerName="manager" Jul 29 15:54:18.884536 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.884437 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" containerName="manager" Jul 29 15:54:18.886748 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.886718 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:18.887654 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.887637 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:18.898871 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.898843 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v"] Jul 29 15:54:18.902544 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.902523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:18.904272 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.904250 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:18.926600 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:18.926575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v"] Jul 29 15:54:18.935827 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.935797 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:18.938240 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.938215 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:18.940600 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.940578 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:18.943089 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:18.943064 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.025647 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.025609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69r4q\" (UniqueName: \"kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.025647 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.025653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.025882 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.025735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnx9l\" (UniqueName: \"kubernetes.io/projected/ffde698a-1f71-4869-aa99-3a04c92ed33b-kube-api-access-wnx9l\") pod \"limitador-operator-controller-manager-859d7b78bd-w797v\" (UID: \"ffde698a-1f71-4869-aa99-3a04c92ed33b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:19.122909 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.122888 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.125495 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.125463 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.126294 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.126200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnx9l\" (UniqueName: \"kubernetes.io/projected/ffde698a-1f71-4869-aa99-3a04c92ed33b-kube-api-access-wnx9l\") pod \"limitador-operator-controller-manager-859d7b78bd-w797v\" (UID: \"ffde698a-1f71-4869-aa99-3a04c92ed33b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:19.126418 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.126319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69r4q\" (UniqueName: \"kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.126418 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.126353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.126743 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.126718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.127685 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.127660 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.136347 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.136324 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.138476 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.138450 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnx9l\" (UniqueName: \"kubernetes.io/projected/ffde698a-1f71-4869-aa99-3a04c92ed33b-kube-api-access-wnx9l\") pod \"limitador-operator-controller-manager-859d7b78bd-w797v\" (UID: \"ffde698a-1f71-4869-aa99-3a04c92ed33b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:19.138719 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.138699 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.139771 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.139749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69r4q\" (UniqueName: \"kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qnzbr\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.141149 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.141129 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.196347 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.196310 2574 generic.go:356] "Generic (PLEG): container finished" podID="189e90a1-c918-41d0-8941-4f64baa24474" containerID="1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2" exitCode=0 Jul 29 15:54:19.196505 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.196360 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.196505 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.196362 2574 scope.go:117] "RemoveContainer" containerID="1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2" Jul 29 15:54:19.197914 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.197887 2574 generic.go:356] "Generic (PLEG): container finished" podID="1df01ba8-53f8-4655-a2ba-7cbea823139e" containerID="be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec" exitCode=0 Jul 29 15:54:19.198021 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.197930 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.199116 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.199089 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.201706 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.201681 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.204961 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.204933 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.206378 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.206307 2574 scope.go:117] "RemoveContainer" containerID="1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2" Jul 29 15:54:19.206922 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.206780 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2\": container with ID starting with 1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2 not found: ID does not exist" containerID="1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2" Jul 29 15:54:19.206922 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.206818 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2"} err="failed to get container status \"1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2\": rpc error: code = NotFound desc = could not find container \"1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2\": container with ID starting with 1eaea4a599a0c3c670dd00c6b4217acc57c03d08e1f7ad856c87df5978c6aba2 not found: ID does not exist" Jul 29 15:54:19.206922 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.206843 2574 scope.go:117] "RemoveContainer" containerID="be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec" Jul 29 15:54:19.208587 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.208556 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.216227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.216211 2574 scope.go:117] "RemoveContainer" containerID="be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec" Jul 29 15:54:19.216480 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.216465 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec\": container with ID starting with be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec not found: ID does not exist" containerID="be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec" Jul 29 15:54:19.216526 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.216487 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec"} err="failed to get container status \"be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec\": rpc error: code = NotFound desc = could not find container \"be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec\": container with ID starting with be08c165270d5b8b9e4f8ce45f480ffdc838256c1ef38810e7ce34c573f4c7ec not found: ID does not exist" Jul 29 15:54:19.226797 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.226777 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume\") pod \"1df01ba8-53f8-4655-a2ba-7cbea823139e\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " Jul 29 15:54:19.226883 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.226829 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mdz\" (UniqueName: \"kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz\") pod \"1df01ba8-53f8-4655-a2ba-7cbea823139e\" (UID: \"1df01ba8-53f8-4655-a2ba-7cbea823139e\") " Jul 29 15:54:19.226883 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.226850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jszf\" (UniqueName: \"kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf\") pod \"189e90a1-c918-41d0-8941-4f64baa24474\" (UID: \"189e90a1-c918-41d0-8941-4f64baa24474\") " Jul 29 15:54:19.227340 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.227317 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1df01ba8-53f8-4655-a2ba-7cbea823139e" (UID: "1df01ba8-53f8-4655-a2ba-7cbea823139e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:54:19.228784 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.228761 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz" (OuterVolumeSpecName: "kube-api-access-g6mdz") pod "1df01ba8-53f8-4655-a2ba-7cbea823139e" (UID: "1df01ba8-53f8-4655-a2ba-7cbea823139e"). InnerVolumeSpecName "kube-api-access-g6mdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:54:19.228864 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.228797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf" (OuterVolumeSpecName: "kube-api-access-5jszf") pod "189e90a1-c918-41d0-8941-4f64baa24474" (UID: "189e90a1-c918-41d0-8941-4f64baa24474"). InnerVolumeSpecName "kube-api-access-5jszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:54:19.280962 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.280926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:19.287349 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.287327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:19.328665 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.328628 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mdz\" (UniqueName: \"kubernetes.io/projected/1df01ba8-53f8-4655-a2ba-7cbea823139e-kube-api-access-g6mdz\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:54:19.328665 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.328660 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jszf\" (UniqueName: \"kubernetes.io/projected/189e90a1-c918-41d0-8941-4f64baa24474-kube-api-access-5jszf\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:54:19.328665 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.328673 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1df01ba8-53f8-4655-a2ba-7cbea823139e-extensions-socket-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:54:19.427965 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.427936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:19.430176 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:54:19.430148 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d47764_13c4_4de8_903f_2859b688ee01.slice/crio-3e364819b815493b2b6a3d5b04600e4062bc69d0912d7f2581781fcfc1a73767 WatchSource:0}: Error finding container 3e364819b815493b2b6a3d5b04600e4062bc69d0912d7f2581781fcfc1a73767: Status 404 returned error can't find the container with id 3e364819b815493b2b6a3d5b04600e4062bc69d0912d7f2581781fcfc1a73767 Jul 29 15:54:19.443839 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.443811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v"] Jul 29 15:54:19.447814 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:54:19.447793 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffde698a_1f71_4869_aa99_3a04c92ed33b.slice/crio-f74000b9ac4ae45f21defc7699b34c767b33b7c9e24c511a304c7bff21bc06d0 WatchSource:0}: Error finding container f74000b9ac4ae45f21defc7699b34c767b33b7c9e24c511a304c7bff21bc06d0: Status 404 returned error can't find the container with id f74000b9ac4ae45f21defc7699b34c767b33b7c9e24c511a304c7bff21bc06d0 Jul 29 15:54:19.509913 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.509885 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.519127 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.519098 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.522087 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.522067 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:19.524298 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:19.524280 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:19.701281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.701210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:19.934565 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:19.934532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:20.205043 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.205011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" event={"ID":"ffde698a-1f71-4869-aa99-3a04c92ed33b","Type":"ContainerStarted","Data":"5547b69dfa191f76478a1006014ee0969896e5b5457127d62cf937c5610c295a"} Jul 29 15:54:20.205231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.205063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" event={"ID":"ffde698a-1f71-4869-aa99-3a04c92ed33b","Type":"ContainerStarted","Data":"f74000b9ac4ae45f21defc7699b34c767b33b7c9e24c511a304c7bff21bc06d0"} Jul 29 15:54:20.205231 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.205181 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:20.206577 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.206554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" event={"ID":"90d47764-13c4-4de8-903f-2859b688ee01","Type":"ContainerStarted","Data":"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246"} Jul 29 15:54:20.206680 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.206596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" event={"ID":"90d47764-13c4-4de8-903f-2859b688ee01","Type":"ContainerStarted","Data":"3e364819b815493b2b6a3d5b04600e4062bc69d0912d7f2581781fcfc1a73767"} Jul 29 15:54:20.206680 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.206627 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:20.207484 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:20.207463 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:20.209721 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:20.209691 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:20.229416 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.229368 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" podStartSLOduration=2.229351823 podStartE2EDuration="2.229351823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:20.228118122 +0000 UTC m=+456.466675332" watchObservedRunningTime="2026-07-29 15:54:20.229351823 +0000 UTC m=+456.467909033" Jul 29 15:54:20.254409 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:20.254372 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rlg8r\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rlg8r" Jul 29 15:54:20.254913 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.254878 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" podStartSLOduration=2.254865102 podStartE2EDuration="2.254865102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:20.252147448 +0000 UTC m=+456.490704658" watchObservedRunningTime="2026-07-29 15:54:20.254865102 +0000 UTC m=+456.493422311" Jul 29 15:54:20.257132 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:20.257108 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9qcj2\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="189e90a1-c918-41d0-8941-4f64baa24474" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9qcj2" Jul 29 15:54:20.333804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.333771 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189e90a1-c918-41d0-8941-4f64baa24474" path="/var/lib/kubelet/pods/189e90a1-c918-41d0-8941-4f64baa24474/volumes" Jul 29 15:54:20.334137 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:20.334123 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df01ba8-53f8-4655-a2ba-7cbea823139e" path="/var/lib/kubelet/pods/1df01ba8-53f8-4655-a2ba-7cbea823139e/volumes" Jul 29 15:54:22.453807 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:22.453765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:22.559282 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:22.559247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:27.284879 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:27.284842 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:27.385774 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:27.385745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:30.836793 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:30.836761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:30.937141 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:30.937107 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:31.214692 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:31.214667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-w797v" Jul 29 15:54:31.214842 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:31.214732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:34.210100 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.210067 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:34.210548 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.210338 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" podUID="90d47764-13c4-4de8-903f-2859b688ee01" containerName="manager" containerID="cri-o://99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246" gracePeriod=10 Jul 29 15:54:34.458144 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.458117 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 15:54:34.462164 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.462117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.467821 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.467799 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:34.484042 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.484014 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 15:54:34.571653 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.571620 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69r4q\" (UniqueName: \"kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q\") pod \"90d47764-13c4-4de8-903f-2859b688ee01\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " Jul 29 15:54:34.571826 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.571676 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume\") pod \"90d47764-13c4-4de8-903f-2859b688ee01\" (UID: \"90d47764-13c4-4de8-903f-2859b688ee01\") " Jul 29 15:54:34.571878 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.571859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpgj\" (UniqueName: \"kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.572058 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.572022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.572120 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.572095 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "90d47764-13c4-4de8-903f-2859b688ee01" (UID: "90d47764-13c4-4de8-903f-2859b688ee01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 15:54:34.573795 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.573769 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q" (OuterVolumeSpecName: "kube-api-access-69r4q") pod "90d47764-13c4-4de8-903f-2859b688ee01" (UID: "90d47764-13c4-4de8-903f-2859b688ee01"). InnerVolumeSpecName "kube-api-access-69r4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:54:34.672650 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.672609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.672825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.672673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpgj\" (UniqueName: \"kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.672825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.672734 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69r4q\" (UniqueName: \"kubernetes.io/projected/90d47764-13c4-4de8-903f-2859b688ee01-kube-api-access-69r4q\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:54:34.672825 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.672743 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d47764-13c4-4de8-903f-2859b688ee01-extensions-socket-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:54:34.673041 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.673023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.684971 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.684950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpgj\" (UniqueName: \"kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj\") pod \"kuadrant-operator-controller-manager-86878c769f-nv676\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.772528 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.772451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:34.933195 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:34.933166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 15:54:34.935047 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:54:34.935020 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a4ea5d_08fe_450c_ae35_6e92a2940ec0.slice/crio-09c3244299becffbb978aba118c3211c7201f4794d5e6d736f36e8404f51f0bd WatchSource:0}: Error finding container 09c3244299becffbb978aba118c3211c7201f4794d5e6d736f36e8404f51f0bd: Status 404 returned error can't find the container with id 09c3244299becffbb978aba118c3211c7201f4794d5e6d736f36e8404f51f0bd Jul 29 15:54:35.279273 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.279238 2574 generic.go:356] "Generic (PLEG): container finished" podID="90d47764-13c4-4de8-903f-2859b688ee01" containerID="99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246" exitCode=0 Jul 29 15:54:35.279675 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.279281 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" Jul 29 15:54:35.279675 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.279319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" event={"ID":"90d47764-13c4-4de8-903f-2859b688ee01","Type":"ContainerDied","Data":"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246"} Jul 29 15:54:35.279675 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.279377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr" event={"ID":"90d47764-13c4-4de8-903f-2859b688ee01","Type":"ContainerDied","Data":"3e364819b815493b2b6a3d5b04600e4062bc69d0912d7f2581781fcfc1a73767"} Jul 29 15:54:35.279675 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.279407 2574 scope.go:117] "RemoveContainer" containerID="99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246" Jul 29 15:54:35.281141 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.281117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" event={"ID":"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0","Type":"ContainerStarted","Data":"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f"} Jul 29 15:54:35.281265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.281155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" event={"ID":"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0","Type":"ContainerStarted","Data":"09c3244299becffbb978aba118c3211c7201f4794d5e6d736f36e8404f51f0bd"} Jul 29 15:54:35.281265 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.281199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:35.288329 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.288314 2574 scope.go:117] "RemoveContainer" containerID="99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246" Jul 29 15:54:35.288593 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:54:35.288575 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246\": container with ID starting with 99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246 not found: ID does not exist" containerID="99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246" Jul 29 15:54:35.288639 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.288600 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246"} err="failed to get container status \"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246\": rpc error: code = NotFound desc = could not find container \"99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246\": container with ID starting with 99b99f885f13802a5b346f8d0e97dfdfcce835e545d1573ab497b3c9ce721246 not found: ID does not exist" Jul 29 15:54:35.311492 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.311409 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" podStartSLOduration=1.311395032 podStartE2EDuration="1.311395032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:35.310312043 +0000 UTC m=+471.548869252" watchObservedRunningTime="2026-07-29 15:54:35.311395032 +0000 UTC m=+471.549952241" Jul 29 15:54:35.323878 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.323852 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:35.333047 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:35.333018 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qnzbr"] Jul 29 15:54:36.333898 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:36.333864 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d47764-13c4-4de8-903f-2859b688ee01" path="/var/lib/kubelet/pods/90d47764-13c4-4de8-903f-2859b688ee01/volumes" Jul 29 15:54:37.504991 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:37.504952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:37.605124 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:37.605092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:40.696736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:40.696700 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:40.796187 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:40.796152 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:43.692996 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:43.692951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:43.792740 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:43.792701 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:45.350811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:45.350767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:45.486517 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:45.486486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:45.580067 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:45.580037 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:46.290076 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:46.290041 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 15:54:50.611892 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.611856 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr"] Jul 29 15:54:50.612282 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.612252 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d47764-13c4-4de8-903f-2859b688ee01" containerName="manager" Jul 29 15:54:50.612282 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.612265 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d47764-13c4-4de8-903f-2859b688ee01" containerName="manager" Jul 29 15:54:50.612356 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.612324 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d47764-13c4-4de8-903f-2859b688ee01" containerName="manager" Jul 29 15:54:50.615119 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.615103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.617659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.617611 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5mj8q\"" Jul 29 15:54:50.645769 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.645744 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr"] Jul 29 15:54:50.657179 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.657150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:50.713259 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713385 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qzk\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-kube-api-access-47qzk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713385 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713385 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713492 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713492 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713492 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713587 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.713587 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.713543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814458 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814458 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814659 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qzk\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-kube-api-access-47qzk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814936 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814936 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814936 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814878 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.814936 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.814908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.815180 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.815078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.815254 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.815234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.815383 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.815364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.817021 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.817005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.817230 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.817212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.826332 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.826304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qzk\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-kube-api-access-47qzk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.827230 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.827207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdbcfec-1562-4f11-ad92-b72b5b6681e4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-68snr\" (UID: \"ecdbcfec-1562-4f11-ad92-b72b5b6681e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.828572 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.828550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:50.925607 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.925523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:50.928753 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:50.928727 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:51.043736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.043708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:54:51.072602 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.072580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr"] Jul 29 15:54:51.074968 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:54:51.074944 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdbcfec_1562_4f11_ad92_b72b5b6681e4.slice/crio-f1cff53e0315ab44dbc2d0a89ab3efabec8e4726c241e260c94f6ddc3569a7a1 WatchSource:0}: Error finding container f1cff53e0315ab44dbc2d0a89ab3efabec8e4726c241e260c94f6ddc3569a7a1: Status 404 returned error can't find the container with id f1cff53e0315ab44dbc2d0a89ab3efabec8e4726c241e260c94f6ddc3569a7a1 Jul 29 15:54:51.076991 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.076952 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:54:51.077086 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.077027 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:54:51.077086 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.077057 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:54:51.355429 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.355397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" event={"ID":"ecdbcfec-1562-4f11-ad92-b72b5b6681e4","Type":"ContainerStarted","Data":"e78c2e1a80eea9e9f68aab2f30d016c89a64a031d7736c63766c63ca94d1bb9e"} Jul 29 15:54:51.355634 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.355449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" event={"ID":"ecdbcfec-1562-4f11-ad92-b72b5b6681e4","Type":"ContainerStarted","Data":"f1cff53e0315ab44dbc2d0a89ab3efabec8e4726c241e260c94f6ddc3569a7a1"} Jul 29 15:54:51.378956 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.378887 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" podStartSLOduration=1.378866266 podStartE2EDuration="1.378866266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:54:51.377720323 +0000 UTC m=+487.616277541" watchObservedRunningTime="2026-07-29 15:54:51.378866266 +0000 UTC m=+487.617423476" Jul 29 15:54:51.925761 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:51.925728 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:52.930706 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:52.930672 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:53.364417 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:53.364385 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:54:53.365410 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:54:53.365392 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:55:02.502270 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:02.502238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:02.600502 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:02.600463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:02.701285 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:02.701252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:03.363274 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:03.363240 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:03.435532 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:03.435495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:23.851366 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:23.851331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:23.941200 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:23.941164 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:24.388086 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:24.388057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:37.846485 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:37.846411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:38.081504 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.081470 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:38.084804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.084789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:38.087638 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.087622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8dff9\"" Jul 29 15:55:38.095309 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.095284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:38.100527 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.100463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:38.128270 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.128241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkmf\" (UniqueName: \"kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf\") pod \"authorino-5d4b67c446-jv89h\" (UID: \"2ec4264b-bbe6-4fe3-9388-6077718a5a0d\") " pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:38.229466 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.229426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkmf\" (UniqueName: \"kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf\") pod \"authorino-5d4b67c446-jv89h\" (UID: \"2ec4264b-bbe6-4fe3-9388-6077718a5a0d\") " pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:38.238267 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.238240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkmf\" (UniqueName: \"kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf\") pod \"authorino-5d4b67c446-jv89h\" (UID: \"2ec4264b-bbe6-4fe3-9388-6077718a5a0d\") " pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:38.278575 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.278539 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:38.278811 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.278800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:38.413198 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.413175 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:38.414730 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:55:38.414699 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec4264b_bbe6_4fe3_9388_6077718a5a0d.slice/crio-94a8d8dfd076355e81b85792f471ccb09dc89da341786ad975673eb205ab8263 WatchSource:0}: Error finding container 94a8d8dfd076355e81b85792f471ccb09dc89da341786ad975673eb205ab8263: Status 404 returned error can't find the container with id 94a8d8dfd076355e81b85792f471ccb09dc89da341786ad975673eb205ab8263 Jul 29 15:55:38.578875 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:38.578837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jv89h" event={"ID":"2ec4264b-bbe6-4fe3-9388-6077718a5a0d","Type":"ContainerStarted","Data":"94a8d8dfd076355e81b85792f471ccb09dc89da341786ad975673eb205ab8263"} Jul 29 15:55:41.598184 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.598100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jv89h" event={"ID":"2ec4264b-bbe6-4fe3-9388-6077718a5a0d","Type":"ContainerStarted","Data":"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d"} Jul 29 15:55:41.598562 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.598187 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jv89h" podUID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" containerName="authorino" containerID="cri-o://28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d" gracePeriod=30 Jul 29 15:55:41.616171 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.616126 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jv89h" podStartSLOduration=0.784136566 podStartE2EDuration="3.616111665s" totalImagesPullingTime="2.831975099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:55:41.61437822 +0000 UTC m=+537.852935434" watchObservedRunningTime="2026-07-29 15:55:41.616111665 +0000 UTC m=+537.854668875" Jul 29 15:55:41.855569 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.855515 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:41.965749 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.965700 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htkmf\" (UniqueName: \"kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf\") pod \"2ec4264b-bbe6-4fe3-9388-6077718a5a0d\" (UID: \"2ec4264b-bbe6-4fe3-9388-6077718a5a0d\") " Jul 29 15:55:41.967798 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:41.967766 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf" (OuterVolumeSpecName: "kube-api-access-htkmf") pod "2ec4264b-bbe6-4fe3-9388-6077718a5a0d" (UID: "2ec4264b-bbe6-4fe3-9388-6077718a5a0d"). InnerVolumeSpecName "kube-api-access-htkmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:55:42.067069 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.067014 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkmf\" (UniqueName: \"kubernetes.io/projected/2ec4264b-bbe6-4fe3-9388-6077718a5a0d-kube-api-access-htkmf\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:55:42.365119 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:55:42.365084 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec4264b_bbe6_4fe3_9388_6077718a5a0d.slice\": RecentStats: unable to find data in memory cache]" Jul 29 15:55:42.604238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.604146 2574 generic.go:356] "Generic (PLEG): container finished" podID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" containerID="28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d" exitCode=0 Jul 29 15:55:42.604238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.604183 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jv89h" Jul 29 15:55:42.604238 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.604226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jv89h" event={"ID":"2ec4264b-bbe6-4fe3-9388-6077718a5a0d","Type":"ContainerDied","Data":"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d"} Jul 29 15:55:42.604786 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.604267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jv89h" event={"ID":"2ec4264b-bbe6-4fe3-9388-6077718a5a0d","Type":"ContainerDied","Data":"94a8d8dfd076355e81b85792f471ccb09dc89da341786ad975673eb205ab8263"} Jul 29 15:55:42.604786 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.604285 2574 scope.go:117] "RemoveContainer" containerID="28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d" Jul 29 15:55:42.612443 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.612424 2574 scope.go:117] "RemoveContainer" containerID="28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d" Jul 29 15:55:42.612688 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:55:42.612668 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d\": container with ID starting with 28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d not found: ID does not exist" containerID="28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d" Jul 29 15:55:42.612730 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.612696 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d"} err="failed to get container status \"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d\": rpc error: code = NotFound desc = could not find container \"28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d\": container with ID starting with 28914cf16f6d6e17445f2bf9eb60afc6b7456e65f9cdb3e83775b24343b1d27d not found: ID does not exist" Jul 29 15:55:42.620736 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.620712 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:42.626559 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:42.626539 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jv89h"] Jul 29 15:55:44.282012 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.281967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:44.335639 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.335608 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" path="/var/lib/kubelet/pods/2ec4264b-bbe6-4fe3-9388-6077718a5a0d/volumes" Jul 29 15:55:44.357905 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.357876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:44.422642 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.422608 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:44.464633 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.464600 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:55:44.465001 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.464968 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" containerName="authorino" Jul 29 15:55:44.465001 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.464994 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" containerName="authorino" Jul 29 15:55:44.465137 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.465059 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec4264b-bbe6-4fe3-9388-6077718a5a0d" containerName="authorino" Jul 29 15:55:44.469461 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.469441 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.472135 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.472115 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vhf4m\"" Jul 29 15:55:44.472228 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.472167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 15:55:44.472228 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.472167 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 15:55:44.479103 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.479079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:55:44.485386 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.485360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.485483 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.485451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwq8\" (UniqueName: \"kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.536356 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.536279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:44.586801 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.586761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.587006 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.586865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwq8\" (UniqueName: \"kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.589236 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.589220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.595479 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.595446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwq8\" (UniqueName: \"kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8\") pod \"maas-controller-7679986d7-ngpxk\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.779705 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.779675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:44.913634 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:44.913600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:55:44.917111 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:55:44.917083 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b468a0_8f1d_4a31_98c9_82c0f68314ef.slice/crio-8359d1b613b0a39a99086dee6129e925f1b6f389d8e1854e2d47130c196b8c8f WatchSource:0}: Error finding container 8359d1b613b0a39a99086dee6129e925f1b6f389d8e1854e2d47130c196b8c8f: Status 404 returned error can't find the container with id 8359d1b613b0a39a99086dee6129e925f1b6f389d8e1854e2d47130c196b8c8f Jul 29 15:55:45.621003 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:45.620957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-ngpxk" event={"ID":"87b468a0-8f1d-4a31-98c9-82c0f68314ef","Type":"ContainerStarted","Data":"8359d1b613b0a39a99086dee6129e925f1b6f389d8e1854e2d47130c196b8c8f"} Jul 29 15:55:47.429963 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:47.429929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:47.526857 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:47.526770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:55:47.633446 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:47.633412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-ngpxk" event={"ID":"87b468a0-8f1d-4a31-98c9-82c0f68314ef","Type":"ContainerStarted","Data":"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493"} Jul 29 15:55:47.633658 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:47.633509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:55:47.659224 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:47.659162 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7679986d7-ngpxk" podStartSLOduration=1.310381517 podStartE2EDuration="3.65913879s" totalImagesPullingTime="2.348757273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:55:47.657881384 +0000 UTC m=+543.896438595" watchObservedRunningTime="2026-07-29 15:55:47.65913879 +0000 UTC m=+543.897696001" Jul 29 15:55:58.641078 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:55:58.641049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:56:05.957506 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:05.957466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:18.343419 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:18.343373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:18.463333 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:18.463296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:19.019808 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:19.019767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:19.199533 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:19.199502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:19.733348 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:19.733321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:19.835052 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:19.835015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:27.819346 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.819314 2574 generic.go:356] "Generic (PLEG): container finished" podID="ecdbcfec-1562-4f11-ad92-b72b5b6681e4" containerID="e78c2e1a80eea9e9f68aab2f30d016c89a64a031d7736c63766c63ca94d1bb9e" exitCode=0 Jul 29 15:56:27.819894 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.819383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" event={"ID":"ecdbcfec-1562-4f11-ad92-b72b5b6681e4","Type":"ContainerDied","Data":"e78c2e1a80eea9e9f68aab2f30d016c89a64a031d7736c63766c63ca94d1bb9e"} Jul 29 15:56:27.819894 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.819791 2574 scope.go:117] "RemoveContainer" containerID="e78c2e1a80eea9e9f68aab2f30d016c89a64a031d7736c63766c63ca94d1bb9e" Jul 29 15:56:27.820749 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.820724 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:56:27.820838 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.820788 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:56:27.820838 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:27.820823 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 15:56:28.832414 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.832381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" event={"ID":"ecdbcfec-1562-4f11-ad92-b72b5b6681e4","Type":"ContainerStarted","Data":"c91d5dd2a5310455ff91b5bcb0c5ae6b3794c3a668a23066d34106036b632fc3"} Jul 29 15:56:28.841528 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.841501 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:28.926503 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.926462 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:56:28.926821 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.926792 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-68snr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 29 15:56:28.926876 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.926852 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" podUID="ecdbcfec-1562-4f11-ad92-b72b5b6681e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 29 15:56:28.948696 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:28.948657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:29.926129 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:29.926089 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-68snr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 29 15:56:29.926513 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:29.926157 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" podUID="ecdbcfec-1562-4f11-ad92-b72b5b6681e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 29 15:56:30.926034 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:30.925998 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-68snr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 29 15:56:30.926217 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:30.926054 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" podUID="ecdbcfec-1562-4f11-ad92-b72b5b6681e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 29 15:56:31.926304 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:31.926263 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-68snr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 29 15:56:31.926706 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:31.926323 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" podUID="ecdbcfec-1562-4f11-ad92-b72b5b6681e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 29 15:56:32.948474 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:32.948437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:56:32.948962 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:32.948930 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:56:32.949819 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:32.949800 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-68snr" Jul 29 15:56:38.943792 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.943763 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl"] Jul 29 15:56:38.946282 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.946254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.949055 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.949027 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 15:56:38.949179 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.949153 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 15:56:38.949341 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.949035 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 15:56:38.950255 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.950238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j2td7\"" Jul 29 15:56:38.957612 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.957590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl"] Jul 29 15:56:38.979546 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.979546 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.979751 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.979751 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.979751 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4d8\" (UniqueName: \"kubernetes.io/projected/5d361af7-dcbb-41f4-9743-1da53a85359b-kube-api-access-pq4d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.979751 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.979720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d361af7-dcbb-41f4-9743-1da53a85359b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:38.983927 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:38.983900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:39.081101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081101 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4d8\" (UniqueName: \"kubernetes.io/projected/5d361af7-dcbb-41f4-9743-1da53a85359b-kube-api-access-pq4d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081375 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d361af7-dcbb-41f4-9743-1da53a85359b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081592 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081632 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.081632 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.081618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.083510 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.083481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d361af7-dcbb-41f4-9743-1da53a85359b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.083634 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.083614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d361af7-dcbb-41f4-9743-1da53a85359b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.089404 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.089384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4d8\" (UniqueName: \"kubernetes.io/projected/5d361af7-dcbb-41f4-9743-1da53a85359b-kube-api-access-pq4d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl\" (UID: \"5d361af7-dcbb-41f4-9743-1da53a85359b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.258689 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.258655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:39.420206 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:56:39.420176 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d361af7_dcbb_41f4_9743_1da53a85359b.slice/crio-c62fae0720b6636e810cf3f4b0f462528795786106371529159a4e80cdb0e9fa WatchSource:0}: Error finding container c62fae0720b6636e810cf3f4b0f462528795786106371529159a4e80cdb0e9fa: Status 404 returned error can't find the container with id c62fae0720b6636e810cf3f4b0f462528795786106371529159a4e80cdb0e9fa Jul 29 15:56:39.420720 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.420693 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl"] Jul 29 15:56:39.890593 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:39.890561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" event={"ID":"5d361af7-dcbb-41f4-9743-1da53a85359b","Type":"ContainerStarted","Data":"c62fae0720b6636e810cf3f4b0f462528795786106371529159a4e80cdb0e9fa"} Jul 29 15:56:42.746527 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.746495 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp"] Jul 29 15:56:42.749996 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.749691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.752644 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.752620 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 15:56:42.761927 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.761899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp"] Jul 29 15:56:42.778430 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.778397 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:42.816054 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4f2\" (UniqueName: \"kubernetes.io/projected/f0e95358-aa84-4c04-85b7-5238e09f51c0-kube-api-access-hw4f2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.816237 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.816237 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.816237 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.816237 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e95358-aa84-4c04-85b7-5238e09f51c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.816444 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.816322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917622 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4f2\" (UniqueName: \"kubernetes.io/projected/f0e95358-aa84-4c04-85b7-5238e09f51c0-kube-api-access-hw4f2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e95358-aa84-4c04-85b7-5238e09f51c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.917832 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.917787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.918227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.918200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.918227 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.918214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.918409 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.918255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.920140 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.920112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e95358-aa84-4c04-85b7-5238e09f51c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.920418 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.920401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e95358-aa84-4c04-85b7-5238e09f51c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:42.926807 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:42.926784 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4f2\" (UniqueName: \"kubernetes.io/projected/f0e95358-aa84-4c04-85b7-5238e09f51c0-kube-api-access-hw4f2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-w5vkp\" (UID: \"f0e95358-aa84-4c04-85b7-5238e09f51c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:43.061616 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:43.061525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:43.205281 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:43.205253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp"] Jul 29 15:56:43.911804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:43.911765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" event={"ID":"f0e95358-aa84-4c04-85b7-5238e09f51c0","Type":"ContainerStarted","Data":"d36005d6e1f95a3eba0381b59bd91af3e8708aadf68df8cddd1e9c865e57aad3"} Jul 29 15:56:44.308550 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:44.308525 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:56:44.308749 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:44.308660 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 15:56:44.918186 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:44.918154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" event={"ID":"f0e95358-aa84-4c04-85b7-5238e09f51c0","Type":"ContainerStarted","Data":"9f3ecb9993bbed5ede576f934bd648fa7b11445ab1ef614a68f078f2c1790e64"} Jul 29 15:56:44.935933 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:44.935878 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" podStartSLOduration=1.828966364 podStartE2EDuration="2.935862062s" totalImagesPullingTime="1.106895698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:44.935367275 +0000 UTC m=+601.173924484" watchObservedRunningTime="2026-07-29 15:56:44.935862062 +0000 UTC m=+601.174419271" Jul 29 15:56:45.924835 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:45.924802 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" event={"ID":"5d361af7-dcbb-41f4-9743-1da53a85359b","Type":"ContainerStarted","Data":"3f6228f2fba9bca85548536244fb537af24b59c301f27c39b3f5d434581534d1"} Jul 29 15:56:45.943319 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:45.943270 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" podStartSLOduration=1.9857284320000002 podStartE2EDuration="7.943253164s" totalImagesPullingTime="5.957524732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:56:45.942312597 +0000 UTC m=+602.180869808" watchObservedRunningTime="2026-07-29 15:56:45.943253164 +0000 UTC m=+602.181810373" Jul 29 15:56:48.985615 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:48.985578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:49.259068 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:49.258958 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:49.259068 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:49.259068 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:49.271524 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:49.271498 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:49.954212 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:49.954177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl" Jul 29 15:56:50.980071 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:50.980037 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:51.117451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:51.117414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:56:53.062300 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:53.062260 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:53.062698 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:53.062333 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:53.075767 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:53.075740 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:53.972651 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:53.972619 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-w5vkp" Jul 29 15:56:57.049817 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:56:57.049788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:57:03.393914 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:03.393812 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:57:05.729726 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:05.729638 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:57:37.806305 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:37.806274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:57:37.903137 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:37.903110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:57:59.812620 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:59.812583 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b595dc6c7-qjhm5"] Jul 29 15:57:59.815451 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:59.815434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:57:59.819211 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:59.819187 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8dff9\"" Jul 29 15:57:59.819292 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:59.819221 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 15:57:59.823875 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:57:59.823854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b595dc6c7-qjhm5"] Jul 29 15:58:00.013707 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.013678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a228d4a1-b1ae-450f-844c-d3e9b6675df6-tls-cert\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.013897 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.013755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ch7\" (UniqueName: \"kubernetes.io/projected/a228d4a1-b1ae-450f-844c-d3e9b6675df6-kube-api-access-c4ch7\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.114547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.114462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a228d4a1-b1ae-450f-844c-d3e9b6675df6-tls-cert\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.114547 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.114526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ch7\" (UniqueName: \"kubernetes.io/projected/a228d4a1-b1ae-450f-844c-d3e9b6675df6-kube-api-access-c4ch7\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.117016 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.116990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a228d4a1-b1ae-450f-844c-d3e9b6675df6-tls-cert\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.124288 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.124264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ch7\" (UniqueName: \"kubernetes.io/projected/a228d4a1-b1ae-450f-844c-d3e9b6675df6-kube-api-access-c4ch7\") pod \"authorino-5b595dc6c7-qjhm5\" (UID: \"a228d4a1-b1ae-450f-844c-d3e9b6675df6\") " pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.124901 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.124886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" Jul 29 15:58:00.202770 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.202722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:00.263650 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.263629 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b595dc6c7-qjhm5"] Jul 29 15:58:00.265857 ip-10-0-130-153 kubenswrapper[2574]: W0729 15:58:00.265823 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda228d4a1_b1ae_450f_844c_d3e9b6675df6.slice/crio-0fe220dec4f453314c19467093ebf503bb3d9a96009fbb632dad6cc39b5810ef WatchSource:0}: Error finding container 0fe220dec4f453314c19467093ebf503bb3d9a96009fbb632dad6cc39b5810ef: Status 404 returned error can't find the container with id 0fe220dec4f453314c19467093ebf503bb3d9a96009fbb632dad6cc39b5810ef Jul 29 15:58:00.267314 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.267291 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 15:58:00.301967 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:00.301921 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:01.278389 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:01.278346 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" event={"ID":"a228d4a1-b1ae-450f-844c-d3e9b6675df6","Type":"ContainerStarted","Data":"a3314721c45554b54108e52b2e6f16c8862f981a1bfec1541dd5d6a788671468"} Jul 29 15:58:01.278880 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:01.278399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" event={"ID":"a228d4a1-b1ae-450f-844c-d3e9b6675df6","Type":"ContainerStarted","Data":"0fe220dec4f453314c19467093ebf503bb3d9a96009fbb632dad6cc39b5810ef"} Jul 29 15:58:01.325465 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:01.325384 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b595dc6c7-qjhm5" podStartSLOduration=1.639863524 podStartE2EDuration="2.32536271s" totalImagesPullingTime="685.499186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 15:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 15:58:01.323516291 +0000 UTC m=+677.562073505" watchObservedRunningTime="2026-07-29 15:58:01.32536271 +0000 UTC m=+677.563919920" Jul 29 15:58:01.339759 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:01.339717 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:01.462460 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:01.462425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:40.130898 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:40.130813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:43.735900 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:43.735863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:44.733562 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:44.733528 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:44.833193 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:44.833159 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:46.999520 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:46.999481 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:58:47.000041 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.000005 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7679986d7-ngpxk" podUID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" containerName="manager" containerID="cri-o://2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493" gracePeriod=30 Jul 29 15:58:47.259640 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.259582 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:58:47.260804 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.260787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert\") pod \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " Jul 29 15:58:47.260962 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.260949 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkwq8\" (UniqueName: \"kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8\") pod \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\" (UID: \"87b468a0-8f1d-4a31-98c9-82c0f68314ef\") " Jul 29 15:58:47.262847 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.262821 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "87b468a0-8f1d-4a31-98c9-82c0f68314ef" (UID: "87b468a0-8f1d-4a31-98c9-82c0f68314ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 15:58:47.262943 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.262888 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8" (OuterVolumeSpecName: "kube-api-access-kkwq8") pod "87b468a0-8f1d-4a31-98c9-82c0f68314ef" (UID: "87b468a0-8f1d-4a31-98c9-82c0f68314ef"). InnerVolumeSpecName "kube-api-access-kkwq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 15:58:47.361647 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.361615 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87b468a0-8f1d-4a31-98c9-82c0f68314ef-webhook-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:58:47.361647 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.361645 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkwq8\" (UniqueName: \"kubernetes.io/projected/87b468a0-8f1d-4a31-98c9-82c0f68314ef-kube-api-access-kkwq8\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 15:58:47.505683 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.505653 2574 generic.go:356] "Generic (PLEG): container finished" podID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" containerID="2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493" exitCode=0 Jul 29 15:58:47.505855 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.505727 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-ngpxk" Jul 29 15:58:47.505855 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.505740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-ngpxk" event={"ID":"87b468a0-8f1d-4a31-98c9-82c0f68314ef","Type":"ContainerDied","Data":"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493"} Jul 29 15:58:47.505855 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.505808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-ngpxk" event={"ID":"87b468a0-8f1d-4a31-98c9-82c0f68314ef","Type":"ContainerDied","Data":"8359d1b613b0a39a99086dee6129e925f1b6f389d8e1854e2d47130c196b8c8f"} Jul 29 15:58:47.505855 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.505839 2574 scope.go:117] "RemoveContainer" containerID="2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493" Jul 29 15:58:47.516197 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.516181 2574 scope.go:117] "RemoveContainer" containerID="2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493" Jul 29 15:58:47.516488 ip-10-0-130-153 kubenswrapper[2574]: E0729 15:58:47.516463 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493\": container with ID starting with 2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493 not found: ID does not exist" containerID="2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493" Jul 29 15:58:47.516536 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.516500 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493"} err="failed to get container status \"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493\": rpc error: code = NotFound desc = could not find container \"2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493\": container with ID starting with 2164e4f909979ff9d223a801fb2639e93fa5781960cf08bbee585484da604493 not found: ID does not exist" Jul 29 15:58:47.530156 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.530130 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:58:47.533525 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:47.533505 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7679986d7-ngpxk"] Jul 29 15:58:48.332739 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:48.332707 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" path="/var/lib/kubelet/pods/87b468a0-8f1d-4a31-98c9-82c0f68314ef/volumes" Jul 29 15:58:55.737544 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:55.737509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:55.842822 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:55.842789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 15:58:55.904800 ip-10-0-130-153 kubenswrapper[2574]: I0729 15:58:55.904771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:00.146152 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.146113 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q"] Jul 29 16:00:00.146640 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.146500 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" containerName="manager" Jul 29 16:00:00.146640 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.146510 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" containerName="manager" Jul 29 16:00:00.146640 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.146567 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b468a0-8f1d-4a31-98c9-82c0f68314ef" containerName="manager" Jul 29 16:00:00.157079 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.157044 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" Jul 29 16:00:00.157884 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.157859 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q"] Jul 29 16:00:00.159823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.159802 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:00:00.169294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.169276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" Jul 29 16:00:00.304714 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.304689 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q"] Jul 29 16:00:00.307512 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:00:00.307486 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc4924d_2a06_4882_9f8e_be88f315bd1c.slice/crio-7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7 WatchSource:0}: Error finding container 7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7: Status 404 returned error can't find the container with id 7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7 Jul 29 16:00:00.865868 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:00.865831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" event={"ID":"2cc4924d-2a06-4882-9f8e-be88f315bd1c","Type":"ContainerStarted","Data":"7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7"} Jul 29 16:00:03.882828 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:03.882738 2574 generic.go:356] "Generic (PLEG): container finished" podID="2cc4924d-2a06-4882-9f8e-be88f315bd1c" containerID="6fdb2045f0d04c372706edf79618ab9a4e82e6117060f77cd40c3240f11ec9bb" exitCode=0 Jul 29 16:00:03.883207 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:03.882823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" event={"ID":"2cc4924d-2a06-4882-9f8e-be88f315bd1c","Type":"ContainerDied","Data":"6fdb2045f0d04c372706edf79618ab9a4e82e6117060f77cd40c3240f11ec9bb"} Jul 29 16:00:05.026419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:05.026389 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" Jul 29 16:00:05.433253 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:05.433220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:05.894764 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:05.894731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" event={"ID":"2cc4924d-2a06-4882-9f8e-be88f315bd1c","Type":"ContainerDied","Data":"7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7"} Jul 29 16:00:05.894764 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:05.894768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755680-4886q" Jul 29 16:00:05.895010 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:05.894787 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd1af57e97711f014e4a7f1f89e3515767e5331c61d232223cfd17245d139c7" Jul 29 16:00:06.649093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.649062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:06.746852 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.746818 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7679986d7-9r2ff"] Jul 29 16:00:06.747216 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.747203 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc4924d-2a06-4882-9f8e-be88f315bd1c" containerName="cleanup" Jul 29 16:00:06.747272 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.747218 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc4924d-2a06-4882-9f8e-be88f315bd1c" containerName="cleanup" Jul 29 16:00:06.747308 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.747274 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc4924d-2a06-4882-9f8e-be88f315bd1c" containerName="cleanup" Jul 29 16:00:06.750376 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.750360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.753219 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.753187 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 16:00:06.754401 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.754364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vhf4m\"" Jul 29 16:00:06.754539 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.754519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 16:00:06.760026 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.760001 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-9r2ff"] Jul 29 16:00:06.856754 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.856716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4rr\" (UniqueName: \"kubernetes.io/projected/ac215dc4-b493-4d54-ab75-28cc0a803fa7-kube-api-access-tl4rr\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.856754 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.856757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac215dc4-b493-4d54-ab75-28cc0a803fa7-webhook-cert\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.957508 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.957483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac215dc4-b493-4d54-ab75-28cc0a803fa7-webhook-cert\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.957661 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.957615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4rr\" (UniqueName: \"kubernetes.io/projected/ac215dc4-b493-4d54-ab75-28cc0a803fa7-kube-api-access-tl4rr\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.959830 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.959804 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac215dc4-b493-4d54-ab75-28cc0a803fa7-webhook-cert\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:06.967359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:06.967337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4rr\" (UniqueName: \"kubernetes.io/projected/ac215dc4-b493-4d54-ab75-28cc0a803fa7-kube-api-access-tl4rr\") pod \"maas-controller-7679986d7-9r2ff\" (UID: \"ac215dc4-b493-4d54-ab75-28cc0a803fa7\") " pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:07.061947 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:07.061907 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:07.202478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:07.202445 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7679986d7-9r2ff"] Jul 29 16:00:07.205733 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:00:07.205702 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac215dc4_b493_4d54_ab75_28cc0a803fa7.slice/crio-d9aa1ae94314417c5b4ccefc4c8f9a494c1cfa3a7b8524c5e8ed19b71a44bc55 WatchSource:0}: Error finding container d9aa1ae94314417c5b4ccefc4c8f9a494c1cfa3a7b8524c5e8ed19b71a44bc55: Status 404 returned error can't find the container with id d9aa1ae94314417c5b4ccefc4c8f9a494c1cfa3a7b8524c5e8ed19b71a44bc55 Jul 29 16:00:07.905292 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:07.905254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9r2ff" event={"ID":"ac215dc4-b493-4d54-ab75-28cc0a803fa7","Type":"ContainerStarted","Data":"d9aa1ae94314417c5b4ccefc4c8f9a494c1cfa3a7b8524c5e8ed19b71a44bc55"} Jul 29 16:00:08.911570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:08.911537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7679986d7-9r2ff" event={"ID":"ac215dc4-b493-4d54-ab75-28cc0a803fa7","Type":"ContainerStarted","Data":"8696f46cd47068851650736bde3d9cac0d99c432cbbee6e7c3afa53901010f86"} Jul 29 16:00:08.911955 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:08.911596 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:08.932420 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:08.932367 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7679986d7-9r2ff" podStartSLOduration=2.01371745 podStartE2EDuration="2.932350992s" totalImagesPullingTime="918.633542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:00:08.930542917 +0000 UTC m=+805.169100120" watchObservedRunningTime="2026-07-29 16:00:08.932350992 +0000 UTC m=+805.170908202" Jul 29 16:00:19.918360 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:19.918326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7679986d7-9r2ff" Jul 29 16:00:47.849387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:47.849358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:49.048304 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:49.048268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:49.148163 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:49.148129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:49.280753 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:49.280723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:55.017925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:55.017888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:55.084693 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:55.084657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:55.179624 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:55.179572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:00:55.372349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:00:55.372254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:01:01.146901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:01:01.146867 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:01:01.247304 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:01:01.247276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:01:10.030236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:01:10.030203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:01:44.339498 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:01:44.339473 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:01:44.341950 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:01:44.341930 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:02:48.948231 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:02:48.948190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:02:50.196094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:02:50.196046 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:03:59.486502 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:03:59.486462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:04:03.121506 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:04:03.121455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:05:14.313226 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:14.313191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 16:05:14.313819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:14.313422 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" podUID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" containerName="manager" containerID="cri-o://06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f" gracePeriod=10 Jul 29 16:05:17.255688 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.255663 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 16:05:17.348943 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.348913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcpgj\" (UniqueName: \"kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj\") pod \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " Jul 29 16:05:17.349157 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.349033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume\") pod \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\" (UID: \"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0\") " Jul 29 16:05:17.349489 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.349458 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" (UID: "d9a4ea5d-08fe-450c-ae35-6e92a2940ec0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:05:17.351003 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.350967 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj" (OuterVolumeSpecName: "kube-api-access-rcpgj") pod "d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" (UID: "d9a4ea5d-08fe-450c-ae35-6e92a2940ec0"). InnerVolumeSpecName "kube-api-access-rcpgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:05:17.401773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.401675 2574 generic.go:356] "Generic (PLEG): container finished" podID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" containerID="06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f" exitCode=0 Jul 29 16:05:17.401773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.401731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" Jul 29 16:05:17.401773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.401742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" event={"ID":"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0","Type":"ContainerDied","Data":"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f"} Jul 29 16:05:17.402089 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.401789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676" event={"ID":"d9a4ea5d-08fe-450c-ae35-6e92a2940ec0","Type":"ContainerDied","Data":"09c3244299becffbb978aba118c3211c7201f4794d5e6d736f36e8404f51f0bd"} Jul 29 16:05:17.402089 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.401814 2574 scope.go:117] "RemoveContainer" containerID="06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f" Jul 29 16:05:17.411653 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.411635 2574 scope.go:117] "RemoveContainer" containerID="06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f" Jul 29 16:05:17.411892 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:05:17.411872 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f\": container with ID starting with 06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f not found: ID does not exist" containerID="06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f" Jul 29 16:05:17.411940 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.411903 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f"} err="failed to get container status \"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f\": rpc error: code = NotFound desc = could not find container \"06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f\": container with ID starting with 06d6107755631c535e0399ebc8049c1a6efca13d3592c7dc1acb6ed4d903327f not found: ID does not exist" Jul 29 16:05:17.436265 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.436230 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 16:05:17.441550 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.441526 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nv676"] Jul 29 16:05:17.450547 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.450524 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpgj\" (UniqueName: \"kubernetes.io/projected/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-kube-api-access-rcpgj\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:05:17.450638 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:17.450547 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0-extensions-socket-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:05:18.333387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:05:18.333356 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" path="/var/lib/kubelet/pods/d9a4ea5d-08fe-450c-ae35-6e92a2940ec0/volumes" Jul 29 16:06:15.513368 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.513298 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs"] Jul 29 16:06:15.513818 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.513651 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" containerName="manager" Jul 29 16:06:15.513818 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.513662 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" containerName="manager" Jul 29 16:06:15.513818 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.513733 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a4ea5d-08fe-450c-ae35-6e92a2940ec0" containerName="manager" Jul 29 16:06:15.516339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.516323 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.521543 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.521522 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j69w2\"" Jul 29 16:06:15.533272 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.533244 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs"] Jul 29 16:06:15.577805 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.577771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.578004 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.577893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgx7\" (UniqueName: \"kubernetes.io/projected/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-kube-api-access-vkgx7\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.678674 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.678642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgx7\" (UniqueName: \"kubernetes.io/projected/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-kube-api-access-vkgx7\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.678846 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.678703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.679153 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.679134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.690412 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.690392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgx7\" (UniqueName: \"kubernetes.io/projected/03bdc93b-ab6b-4744-b05e-36368a1f0bfb-kube-api-access-vkgx7\") pod \"kuadrant-operator-controller-manager-86878c769f-knrzs\" (UID: \"03bdc93b-ab6b-4744-b05e-36368a1f0bfb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.826275 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.826170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:15.967170 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:15.967142 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs"] Jul 29 16:06:15.968828 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:06:15.968803 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bdc93b_ab6b_4744_b05e_36368a1f0bfb.slice/crio-8b3f76a53beb38644d266d22dcda14f258e9e4e07a560a300e6b38ef7375ac67 WatchSource:0}: Error finding container 8b3f76a53beb38644d266d22dcda14f258e9e4e07a560a300e6b38ef7375ac67: Status 404 returned error can't find the container with id 8b3f76a53beb38644d266d22dcda14f258e9e4e07a560a300e6b38ef7375ac67 Jul 29 16:06:16.679399 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:16.679352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" event={"ID":"03bdc93b-ab6b-4744-b05e-36368a1f0bfb","Type":"ContainerStarted","Data":"f66e0e2942e8e1899c3a81d60c6a8aad01b9b2ff4854048f828eed8b79a41745"} Jul 29 16:06:16.679772 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:16.679411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" event={"ID":"03bdc93b-ab6b-4744-b05e-36368a1f0bfb","Type":"ContainerStarted","Data":"8b3f76a53beb38644d266d22dcda14f258e9e4e07a560a300e6b38ef7375ac67"} Jul 29 16:06:16.679772 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:16.679432 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:16.707067 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:16.707018 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" podStartSLOduration=1.707000509 podStartE2EDuration="1.707000509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:06:16.706551944 +0000 UTC m=+1172.945109159" watchObservedRunningTime="2026-07-29 16:06:16.707000509 +0000 UTC m=+1172.945557718" Jul 29 16:06:27.687966 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:27.687935 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knrzs" Jul 29 16:06:44.367783 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:44.367661 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:06:44.382443 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:06:44.379083 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:09:06.981632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:09:06.981543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:09:14.678691 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:09:14.678656 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:09:28.307092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:09:28.307052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:09:59.608524 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:09:59.608493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:10:01.450157 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:10:01.450127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:11:09.516902 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:11:09.516874 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:11:09.622561 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:11:09.622528 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:11:44.399814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:11:44.399708 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:11:44.410888 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:11:44.410865 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:12:16.484365 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:12:16.484273 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:13.816852 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:13.816818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:35.518568 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:35.518485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:38.620204 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:38.620170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:38.721299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:38.721268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:39.133042 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:39.133007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:45.580387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:45.580354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:45.668255 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:45.668220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:45.812307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:45.812263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:46.136666 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.136633 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:13:46.140146 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.140128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.143859 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.143842 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2f353f9d-openshift-default-dockercfg-knbnt\"" Jul 29 16:13:46.164806 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.164765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:46.166865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.166840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:13:46.260120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.260088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:46.290036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmphr\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.290307 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.290290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.390936 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.390936 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.390936 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.390936 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.390974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmphr\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391585 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391585 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391585 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391585 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.391933 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.391908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.393562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.393529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.394011 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.393968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.400765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.400732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.401160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.401137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmphr\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr\") pod \"e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.450393 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.450353 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:46.605250 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.605223 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:13:46.607269 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:13:46.607238 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478ec499_7393_4fbc_9dc6_b97f82d7378e.slice/crio-9921d8bbad36e1c025239a8a37df4ae5d29c2bbd5423a0d3ef3465f20093bb70 WatchSource:0}: Error finding container 9921d8bbad36e1c025239a8a37df4ae5d29c2bbd5423a0d3ef3465f20093bb70: Status 404 returned error can't find the container with id 9921d8bbad36e1c025239a8a37df4ae5d29c2bbd5423a0d3ef3465f20093bb70 Jul 29 16:13:46.609520 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.609486 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:13:46.609628 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.609571 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:13:46.609628 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.609614 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:13:46.901516 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.901416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerStarted","Data":"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9"} Jul 29 16:13:46.901516 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.901474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerStarted","Data":"9921d8bbad36e1c025239a8a37df4ae5d29c2bbd5423a0d3ef3465f20093bb70"} Jul 29 16:13:46.936282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:46.936230 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podStartSLOduration=0.936213726 podStartE2EDuration="936.213726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:13:46.934159159 +0000 UTC m=+1623.172716371" watchObservedRunningTime="2026-07-29 16:13:46.936213726 +0000 UTC m=+1623.174770936" Jul 29 16:13:47.450881 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:47.450837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:47.456577 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:47.456550 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:47.908189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:47.908158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:47.909313 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:47.909283 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:13:48.353747 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:48.353709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:48.505925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:48.505891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:48.600310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:48.600269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:48.758348 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:48.758318 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:49.147582 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.147472 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:49.253276 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.253245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:13:49.285705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.285670 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:13:49.289427 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.289397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.292595 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.292568 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2f353f9d\"" Jul 29 16:13:49.293006 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.292647 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2f353f9d-dockercfg-jv99j\"" Jul 29 16:13:49.304401 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.304372 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:13:49.421328 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.421231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qk84\" (UniqueName: \"kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.421328 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.421293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.522382 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.522340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk84\" (UniqueName: \"kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.522594 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.522395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.523145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.523125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.533777 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.533743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk84\" (UniqueName: \"kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84\") pod \"payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.605182 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.605147 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:49.765655 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.765629 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:13:49.767795 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:13:49.767759 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eafb979_7f54_40e9_8fe5_2b487235c7ab.slice/crio-8172a4533a7ad1cbdaf79ef6be1433603902417e8af720111bc3c0b295e53b3b WatchSource:0}: Error finding container 8172a4533a7ad1cbdaf79ef6be1433603902417e8af720111bc3c0b295e53b3b: Status 404 returned error can't find the container with id 8172a4533a7ad1cbdaf79ef6be1433603902417e8af720111bc3c0b295e53b3b Jul 29 16:13:49.769895 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.769877 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:13:49.919780 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:49.919747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" event={"ID":"4eafb979-7f54-40e9-8fe5-2b487235c7ab","Type":"ContainerStarted","Data":"8172a4533a7ad1cbdaf79ef6be1433603902417e8af720111bc3c0b295e53b3b"} Jul 29 16:13:52.940229 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:52.940193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" event={"ID":"4eafb979-7f54-40e9-8fe5-2b487235c7ab","Type":"ContainerStarted","Data":"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c"} Jul 29 16:13:52.940627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:52.940270 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:13:52.960261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:13:52.960210 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" podStartSLOduration=1.587357632 podStartE2EDuration="3.960193602s" totalImagesPullingTime="2.37283597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:13:52.959592455 +0000 UTC m=+1629.198149664" watchObservedRunningTime="2026-07-29 16:13:52.960193602 +0000 UTC m=+1629.198750811" Jul 29 16:14:02.909544 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.909507 2574 patch_prober.go:28] interesting pod/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 16:14:02.910191 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.909571 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 16:14:02.995729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.995697 2574 generic.go:356] "Generic (PLEG): container finished" podID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerID="a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9" exitCode=0 Jul 29 16:14:02.995899 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.995739 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerDied","Data":"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9"} Jul 29 16:14:02.996177 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.996160 2574 scope.go:117] "RemoveContainer" containerID="a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9" Jul 29 16:14:02.997292 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.997261 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:02.997374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.997340 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:02.997413 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:02.997375 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:03.949218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:03.949188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:14:04.022321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:04.022282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerStarted","Data":"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83"} Jul 29 16:14:04.451408 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:04.451363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:04.451686 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:04.451660 2574 patch_prober.go:28] interesting pod/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 16:14:04.451780 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:04.451717 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 16:14:05.451696 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:05.451658 2574 patch_prober.go:28] interesting pod/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 16:14:05.452125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:05.451728 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 16:14:06.450780 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:06.450745 2574 patch_prober.go:28] interesting pod/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 16:14:06.450953 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:06.450815 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 16:14:07.450855 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:07.450815 2574 patch_prober.go:28] interesting pod/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 16:14:07.451275 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:07.450891 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 16:14:08.454604 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:08.454573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:08.455069 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:08.454844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:08.455626 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:08.455611 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:29.163087 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.163036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:29.194460 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.194271 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:14:29.194669 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.194539 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" podUID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" containerName="payload-processing" containerID="cri-o://f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c" gracePeriod=30 Jul 29 16:14:29.474565 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.474538 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:14:29.557094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.557064 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:14:29.557299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.557276 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" containerID="cri-o://a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83" gracePeriod=30 Jul 29 16:14:29.621578 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.621542 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume\") pod \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " Jul 29 16:14:29.621758 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.621611 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qk84\" (UniqueName: \"kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84\") pod \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\" (UID: \"4eafb979-7f54-40e9-8fe5-2b487235c7ab\") " Jul 29 16:14:29.622040 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.622016 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4eafb979-7f54-40e9-8fe5-2b487235c7ab" (UID: "4eafb979-7f54-40e9-8fe5-2b487235c7ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:14:29.623746 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.623723 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84" (OuterVolumeSpecName: "kube-api-access-8qk84") pod "4eafb979-7f54-40e9-8fe5-2b487235c7ab" (UID: "4eafb979-7f54-40e9-8fe5-2b487235c7ab"). InnerVolumeSpecName "kube-api-access-8qk84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:14:29.723332 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.723297 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eafb979-7f54-40e9-8fe5-2b487235c7ab-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:29.723332 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:29.723328 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qk84\" (UniqueName: \"kubernetes.io/projected/4eafb979-7f54-40e9-8fe5-2b487235c7ab-kube-api-access-8qk84\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:30.149813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.149717 2574 generic.go:356] "Generic (PLEG): container finished" podID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" containerID="f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c" exitCode=0 Jul 29 16:14:30.149813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.149769 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" Jul 29 16:14:30.150055 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.149813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" event={"ID":"4eafb979-7f54-40e9-8fe5-2b487235c7ab","Type":"ContainerDied","Data":"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c"} Jul 29 16:14:30.150055 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.149862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf" event={"ID":"4eafb979-7f54-40e9-8fe5-2b487235c7ab","Type":"ContainerDied","Data":"8172a4533a7ad1cbdaf79ef6be1433603902417e8af720111bc3c0b295e53b3b"} Jul 29 16:14:30.150055 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.149891 2574 scope.go:117] "RemoveContainer" containerID="f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c" Jul 29 16:14:30.159151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.159132 2574 scope.go:117] "RemoveContainer" containerID="f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c" Jul 29 16:14:30.159409 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:14:30.159389 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c\": container with ID starting with f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c not found: ID does not exist" containerID="f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c" Jul 29 16:14:30.159459 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.159420 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c"} err="failed to get container status \"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c\": rpc error: code = NotFound desc = could not find container \"f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c\": container with ID starting with f80a14da809db511783c10c8c92d6bbebb98c9a7817f7c91be826346b4e3ea6c not found: ID does not exist" Jul 29 16:14:30.177952 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.177915 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:14:30.187953 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.187921 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f353f9d-7fcb5b787b-h7tzf"] Jul 29 16:14:30.333434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.333402 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" path="/var/lib/kubelet/pods/4eafb979-7f54-40e9-8fe5-2b487235c7ab/volumes" Jul 29 16:14:30.519898 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.519867 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:30.650785 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.650748 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:14:30.651356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.651322 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" containerName="payload-processing" Jul 29 16:14:30.651356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.651350 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" containerName="payload-processing" Jul 29 16:14:30.651579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.651461 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eafb979-7f54-40e9-8fe5-2b487235c7ab" containerName="payload-processing" Jul 29 16:14:30.655902 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.655878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.659747 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.659724 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-959c7923-gw-openshift-default-dockercfg-65mxs\"" Jul 29 16:14:30.675078 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.675040 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:14:30.695175 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.695132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:30.733303 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733303 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrlj\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.733540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.733486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrlj\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834836 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834627 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.834836 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.834652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.835139 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.835113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:30.835139 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.835143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.835321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.835145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.835321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.835311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.837034 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.836999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.837197 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.837174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.843740 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.843712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrlj\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.849589 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.849563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token\") pod \"e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:30.966995 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:30.966944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:31.116634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:31.116485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:14:31.119147 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:14:31.119111 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb5c544_a384_460a_9302_a16d6b6e442a.slice/crio-a8446730394c2811e2b75f85bc89cb6e8f869d5fc5d18641d61dbcc03bb89450 WatchSource:0}: Error finding container a8446730394c2811e2b75f85bc89cb6e8f869d5fc5d18641d61dbcc03bb89450: Status 404 returned error can't find the container with id a8446730394c2811e2b75f85bc89cb6e8f869d5fc5d18641d61dbcc03bb89450 Jul 29 16:14:31.121265 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:31.121233 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:31.121351 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:31.121323 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:31.121394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:31.121355 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:31.156028 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:31.155971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerStarted","Data":"a8446730394c2811e2b75f85bc89cb6e8f869d5fc5d18641d61dbcc03bb89450"} Jul 29 16:14:32.163427 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:32.163391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerStarted","Data":"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80"} Jul 29 16:14:32.190181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:32.190126 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podStartSLOduration=2.190108654 podStartE2EDuration="2.190108654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:14:32.188413571 +0000 UTC m=+1668.426970781" watchObservedRunningTime="2026-07-29 16:14:32.190108654 +0000 UTC m=+1668.428665864" Jul 29 16:14:32.967485 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:32.967446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:32.972753 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:32.972723 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:33.168781 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:33.168750 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:33.169767 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:33.169744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:33.706915 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:33.706870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:33.836595 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:33.836556 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:34.031108 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.031009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:34.420512 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.420378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:34.521494 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.521454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:14:34.830326 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.830293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:34.971125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971140 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971169 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971200 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmphr\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971232 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971284 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971594 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971350 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971594 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971594 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971495 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs\") pod \"478ec499-7393-4fbc-9dc6-b97f82d7378e\" (UID: \"478ec499-7393-4fbc-9dc6-b97f82d7378e\") " Jul 29 16:14:34.971740 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:14:34.971799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971786 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:34.971892 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:14:34.971949 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.971917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:14:34.972166 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.972144 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:14:34.972283 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.972163 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data" (OuterVolumeSpecName: "istio-data") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:14:34.973988 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.973949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:14:34.974195 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.974164 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:14:34.974289 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.974202 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token" (OuterVolumeSpecName: "istio-token") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:14:34.974289 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:34.974231 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr" (OuterVolumeSpecName: "kube-api-access-zmphr") pod "478ec499-7393-4fbc-9dc6-b97f82d7378e" (UID: "478ec499-7393-4fbc-9dc6-b97f82d7378e"). InnerVolumeSpecName "kube-api-access-zmphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:14:35.072323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072281 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072312 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072324 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmphr\" (UniqueName: \"kubernetes.io/projected/478ec499-7393-4fbc-9dc6-b97f82d7378e-kube-api-access-zmphr\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072336 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478ec499-7393-4fbc-9dc6-b97f82d7378e-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072350 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072360 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072376 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.072579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.072386 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478ec499-7393-4fbc-9dc6-b97f82d7378e-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:14:35.179648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.179616 2574 generic.go:356] "Generic (PLEG): container finished" podID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerID="a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83" exitCode=0 Jul 29 16:14:35.179826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.179673 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" Jul 29 16:14:35.179826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.179711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerDied","Data":"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83"} Jul 29 16:14:35.179826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.179766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p" event={"ID":"478ec499-7393-4fbc-9dc6-b97f82d7378e","Type":"ContainerDied","Data":"9921d8bbad36e1c025239a8a37df4ae5d29c2bbd5423a0d3ef3465f20093bb70"} Jul 29 16:14:35.179826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.179792 2574 scope.go:117] "RemoveContainer" containerID="a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83" Jul 29 16:14:35.193100 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.192330 2574 scope.go:117] "RemoveContainer" containerID="a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9" Jul 29 16:14:35.203861 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.203828 2574 scope.go:117] "RemoveContainer" containerID="a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83" Jul 29 16:14:35.204251 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:14:35.204231 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83\": container with ID starting with a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83 not found: ID does not exist" containerID="a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83" Jul 29 16:14:35.204388 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.204262 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83"} err="failed to get container status \"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83\": rpc error: code = NotFound desc = could not find container \"a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83\": container with ID starting with a89956f48105d3c5e60b55e3bec9cbbb7eb1c4c423ad9881b60c8df6fd70bf83 not found: ID does not exist" Jul 29 16:14:35.204388 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.204281 2574 scope.go:117] "RemoveContainer" containerID="a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9" Jul 29 16:14:35.204613 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:14:35.204585 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9\": container with ID starting with a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9 not found: ID does not exist" containerID="a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9" Jul 29 16:14:35.204694 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.204618 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9"} err="failed to get container status \"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9\": rpc error: code = NotFound desc = could not find container \"a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9\": container with ID starting with a1abf31be0f7e7a0c54e39ef75d5d60e7828b817715cb362bbcd95c2963ba3c9 not found: ID does not exist" Jul 29 16:14:35.207020 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.206998 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:14:35.210439 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:35.210408 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2f353f9d-openshift-default-5785bbb6cb-6bq2p"] Jul 29 16:14:36.334190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:36.334157 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" path="/var/lib/kubelet/pods/478ec499-7393-4fbc-9dc6-b97f82d7378e/volumes" Jul 29 16:14:48.170632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.170596 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 29 16:14:48.171087 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.170668 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 29 16:14:48.254286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.254254 2574 generic.go:356] "Generic (PLEG): container finished" podID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerID="2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80" exitCode=0 Jul 29 16:14:48.254458 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.254303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerDied","Data":"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80"} Jul 29 16:14:48.254704 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.254690 2574 scope.go:117] "RemoveContainer" containerID="2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80" Jul 29 16:14:48.255772 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.255743 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:48.255846 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.255831 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:48.255882 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:48.255869 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:14:49.261450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:49.261414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerStarted","Data":"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e"} Jul 29 16:14:49.968106 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:49.968070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:49.968380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:49.968357 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 29 16:14:49.968470 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:49.968412 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 29 16:14:50.968362 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:50.968323 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 29 16:14:50.968853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:50.968402 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 29 16:14:51.967527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:51.967486 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 29 16:14:51.967715 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:51.967561 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 29 16:14:52.967712 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:52.967674 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 29 16:14:52.968149 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:52.967748 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 29 16:14:53.972736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:53.972698 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:53.973232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:53.973093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:53.974694 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:53.974670 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:14:58.663464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.663424 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt"] Jul 29 16:14:58.663859 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.663815 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:14:58.663859 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.663825 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:14:58.663943 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.663891 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:14:58.667296 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.667279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:14:58.670073 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.670042 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:14:58.670210 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.670172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:14:58.676269 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.676247 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt"] Jul 29 16:14:58.691906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.691871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt\" (UID: \"10d3144d-4516-4e1b-b505-455327f2ecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:14:58.793105 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.793069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt\" (UID: \"10d3144d-4516-4e1b-b505-455327f2ecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:14:58.793720 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.793699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt\" (UID: \"10d3144d-4516-4e1b-b505-455327f2ecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:14:58.979265 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:58.979234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:14:59.332337 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:14:59.332307 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt"] Jul 29 16:14:59.333925 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:14:59.333903 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d3144d_4516_4e1b_b505_455327f2ecbe.slice/crio-d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973 WatchSource:0}: Error finding container d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973: Status 404 returned error can't find the container with id d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973 Jul 29 16:15:00.138109 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.138075 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2"] Jul 29 16:15:00.138498 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.138481 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:15:00.138498 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.138497 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:15:00.138574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.138569 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="478ec499-7393-4fbc-9dc6-b97f82d7378e" containerName="istio-proxy" Jul 29 16:15:00.141745 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.141724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" Jul 29 16:15:00.151201 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.151174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2"] Jul 29 16:15:00.162950 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.162932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" Jul 29 16:15:00.305663 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.305639 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2"] Jul 29 16:15:00.307605 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:15:00.307554 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbafa0b_195a_4f7c_a7db_bb592208a5a3.slice/crio-11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f WatchSource:0}: Error finding container 11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f: Status 404 returned error can't find the container with id 11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f Jul 29 16:15:00.318006 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.317959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" event={"ID":"2dbafa0b-195a-4f7c-a7db-bb592208a5a3","Type":"ContainerStarted","Data":"11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f"} Jul 29 16:15:00.319434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.319414 2574 generic.go:356] "Generic (PLEG): container finished" podID="10d3144d-4516-4e1b-b505-455327f2ecbe" containerID="73339136c60a4124cb793d15774c05eecf0e32e1dcb3bd7b273e5c14393c2bf8" exitCode=0 Jul 29 16:15:00.319528 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.319497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" event={"ID":"10d3144d-4516-4e1b-b505-455327f2ecbe","Type":"ContainerDied","Data":"73339136c60a4124cb793d15774c05eecf0e32e1dcb3bd7b273e5c14393c2bf8"} Jul 29 16:15:00.319571 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:00.319546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" event={"ID":"10d3144d-4516-4e1b-b505-455327f2ecbe","Type":"ContainerStarted","Data":"d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973"} Jul 29 16:15:01.325710 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.325672 2574 generic.go:356] "Generic (PLEG): container finished" podID="2dbafa0b-195a-4f7c-a7db-bb592208a5a3" containerID="3b0d3921fcd72801bc71664a54898e3a6c86e443d44b750c2b80138447f0d1de" exitCode=0 Jul 29 16:15:01.326118 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.325762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" event={"ID":"2dbafa0b-195a-4f7c-a7db-bb592208a5a3","Type":"ContainerDied","Data":"3b0d3921fcd72801bc71664a54898e3a6c86e443d44b750c2b80138447f0d1de"} Jul 29 16:15:01.469228 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.469200 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:15:01.520239 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.520205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca\") pod \"10d3144d-4516-4e1b-b505-455327f2ecbe\" (UID: \"10d3144d-4516-4e1b-b505-455327f2ecbe\") " Jul 29 16:15:01.520558 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.520534 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10d3144d-4516-4e1b-b505-455327f2ecbe" (UID: "10d3144d-4516-4e1b-b505-455327f2ecbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:01.621448 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:01.621408 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d3144d-4516-4e1b-b505-455327f2ecbe-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:02.338334 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:02.338304 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" Jul 29 16:15:02.338786 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:02.338642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt" event={"ID":"10d3144d-4516-4e1b-b505-455327f2ecbe","Type":"ContainerDied","Data":"d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973"} Jul 29 16:15:02.338786 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:02.338690 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b22e53321a2eb990e03b4e2a09ca6cca6a5a02b94361d23a68b4b66a07a973" Jul 29 16:15:02.476348 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:02.476319 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" Jul 29 16:15:03.344399 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:03.344370 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" Jul 29 16:15:03.344847 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:03.344369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755695-72jj2" event={"ID":"2dbafa0b-195a-4f7c-a7db-bb592208a5a3","Type":"ContainerDied","Data":"11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f"} Jul 29 16:15:03.344847 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:03.344497 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e4c0508fe6099de4b708a125964f8668ebcffdc75e66fad689748f3a294b2f" Jul 29 16:15:05.451078 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.451040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:05.711281 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.711244 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:05.711689 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.711667 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt"] Jul 29 16:15:05.715863 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.715837 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-959c7923-dkbgt"] Jul 29 16:15:05.805491 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.805457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:05.847030 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.846997 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:15:05.847227 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.847205 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" containerID="cri-o://53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e" gracePeriod=30 Jul 29 16:15:05.898809 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:05.898777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:06.079453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.079373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:06.321569 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.321534 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:06.332731 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.332665 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d3144d-4516-4e1b-b505-455327f2ecbe" path="/var/lib/kubelet/pods/10d3144d-4516-4e1b-b505-455327f2ecbe/volumes" Jul 29 16:15:06.604274 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604194 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:06.604634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604584 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbafa0b-195a-4f7c-a7db-bb592208a5a3" containerName="cleanup" Jul 29 16:15:06.604634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604597 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbafa0b-195a-4f7c-a7db-bb592208a5a3" containerName="cleanup" Jul 29 16:15:06.604634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604609 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d3144d-4516-4e1b-b505-455327f2ecbe" containerName="revoke-keys" Jul 29 16:15:06.604634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604615 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d3144d-4516-4e1b-b505-455327f2ecbe" containerName="revoke-keys" Jul 29 16:15:06.604803 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604680 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbafa0b-195a-4f7c-a7db-bb592208a5a3" containerName="cleanup" Jul 29 16:15:06.604803 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.604690 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d3144d-4516-4e1b-b505-455327f2ecbe" containerName="revoke-keys" Jul 29 16:15:06.608096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.608079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.620535 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.620516 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d49b3458-openshift-default-dockercfg-5jmp5\"" Jul 29 16:15:06.665026 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.664953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665204 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665204 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9kf\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665556 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.665556 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.665377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.668525 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.668500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:06.672533 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.672503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:06.766124 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9kf\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766647 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766647 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.766888 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.766838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.767086 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.767070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.768582 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.768564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.768776 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.768759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.778341 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.778313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9kf\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.778622 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.778603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token\") pod \"e2e-ait-d49b3458-openshift-default-7685497b6-f8v98\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.917749 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.917660 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:06.922912 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:06.922884 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:07.085394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.085368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:07.087255 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:15:07.087230 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11e6bd9_38fc_44de_b873_8c5b432831b9.slice/crio-4746d4dab57df6245c04adeb739a2a6541c7c74e83446da89a6257b0c7c271c7 WatchSource:0}: Error finding container 4746d4dab57df6245c04adeb739a2a6541c7c74e83446da89a6257b0c7c271c7: Status 404 returned error can't find the container with id 4746d4dab57df6245c04adeb739a2a6541c7c74e83446da89a6257b0c7c271c7 Jul 29 16:15:07.089356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.089325 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:07.089446 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.089386 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:07.089446 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.089413 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:07.363808 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.363771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerStarted","Data":"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913"} Jul 29 16:15:07.364022 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.363834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerStarted","Data":"4746d4dab57df6245c04adeb739a2a6541c7c74e83446da89a6257b0c7c271c7"} Jul 29 16:15:07.398775 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.398712 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podStartSLOduration=1.398690753 podStartE2EDuration="1.398690753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:15:07.396583918 +0000 UTC m=+1703.635141139" watchObservedRunningTime="2026-07-29 16:15:07.398690753 +0000 UTC m=+1703.637247964" Jul 29 16:15:07.918211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:07.918179 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:08.801225 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:08.801186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:08.901494 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:08.901464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:08.923038 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:08.923011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:08.975305 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:08.975275 2574 patch_prober.go:28] interesting pod/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:15:08.975463 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:08.975323 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:15:09.008625 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.008594 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:09.107434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.107353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:09.379183 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.379095 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:09.380088 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.380066 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:09.640134 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.640047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:09.873815 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:09.873785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:11.110837 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.110778 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:15:11.213567 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213604 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213645 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213707 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213753 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhrlj\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213829 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213856 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket\") pod \"7eb5c544-a384-460a-9302-a16d6b6e442a\" (UID: \"7eb5c544-a384-460a-9302-a16d6b6e442a\") " Jul 29 16:15:11.213957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.213864 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:11.214229 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214160 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:11.214279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214246 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.214279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214266 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb5c544-a384-460a-9302-a16d6b6e442a-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.214380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214309 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:11.214380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:11.214484 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.214446 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:11.216016 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.215966 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:11.216280 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.216259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:15:11.216280 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.216271 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj" (OuterVolumeSpecName: "kube-api-access-lhrlj") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "kube-api-access-lhrlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:11.216402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.216280 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eb5c544-a384-460a-9302-a16d6b6e442a" (UID: "7eb5c544-a384-460a-9302-a16d6b6e442a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:11.315120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315091 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315116 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315124 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315133 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315141 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrlj\" (UniqueName: \"kubernetes.io/projected/7eb5c544-a384-460a-9302-a16d6b6e442a-kube-api-access-lhrlj\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315150 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.315346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.315159 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb5c544-a384-460a-9302-a16d6b6e442a-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:11.390218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.390135 2574 generic.go:356] "Generic (PLEG): container finished" podID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerID="53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e" exitCode=0 Jul 29 16:15:11.390218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.390182 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" Jul 29 16:15:11.390432 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.390220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerDied","Data":"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e"} Jul 29 16:15:11.390432 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.390270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7" event={"ID":"7eb5c544-a384-460a-9302-a16d6b6e442a","Type":"ContainerDied","Data":"a8446730394c2811e2b75f85bc89cb6e8f869d5fc5d18641d61dbcc03bb89450"} Jul 29 16:15:11.390432 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.390292 2574 scope.go:117] "RemoveContainer" containerID="53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e" Jul 29 16:15:11.400909 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.400890 2574 scope.go:117] "RemoveContainer" containerID="2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80" Jul 29 16:15:11.409160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.409142 2574 scope.go:117] "RemoveContainer" containerID="53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e" Jul 29 16:15:11.409412 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:15:11.409396 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e\": container with ID starting with 53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e not found: ID does not exist" containerID="53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e" Jul 29 16:15:11.409466 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.409419 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e"} err="failed to get container status \"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e\": rpc error: code = NotFound desc = could not find container \"53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e\": container with ID starting with 53c842d2903d5e88a407b231f83384ad667ff674de97499468852ef65fe1049e not found: ID does not exist" Jul 29 16:15:11.409466 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.409434 2574 scope.go:117] "RemoveContainer" containerID="2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80" Jul 29 16:15:11.409672 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:15:11.409658 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80\": container with ID starting with 2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80 not found: ID does not exist" containerID="2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80" Jul 29 16:15:11.409710 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.409676 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80"} err="failed to get container status \"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80\": rpc error: code = NotFound desc = could not find container \"2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80\": container with ID starting with 2e5443f600414fe925c31ef9eac10a1181130c277f67e4414da4eea80ce68e80 not found: ID does not exist" Jul 29 16:15:11.419685 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.419659 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:15:11.421542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:11.421520 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-959c7923-gw-openshift-default-5f9575c9b-h6pq7"] Jul 29 16:15:12.332870 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:12.332839 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" path="/var/lib/kubelet/pods/7eb5c544-a384-460a-9302-a16d6b6e442a/volumes" Jul 29 16:15:24.380825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.380786 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 29 16:15:24.381289 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.380849 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 29 16:15:24.461068 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.461038 2574 generic.go:356] "Generic (PLEG): container finished" podID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerID="92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913" exitCode=0 Jul 29 16:15:24.461251 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.461118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerDied","Data":"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913"} Jul 29 16:15:24.461534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.461520 2574 scope.go:117] "RemoveContainer" containerID="92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913" Jul 29 16:15:24.462606 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.462580 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:24.462671 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.462650 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:24.462718 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:24.462681 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:25.468050 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:25.468016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerStarted","Data":"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b"} Jul 29 16:15:25.919225 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:25.919123 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:25.919225 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:25.919194 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 29 16:15:25.919480 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:25.919271 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 29 16:15:26.918511 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:26.918476 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 29 16:15:26.918904 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:26.918537 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 29 16:15:27.918650 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:27.918599 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 29 16:15:27.919084 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:27.918683 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 29 16:15:28.918330 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:28.918287 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 29 16:15:28.918523 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:28.918363 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 29 16:15:29.922108 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:29.922075 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:29.922510 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:29.922488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:29.923287 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:29.923261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:34.550438 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.550397 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4"] Jul 29 16:15:34.550841 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.550798 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:34.550841 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.550815 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:34.550924 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.550912 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:34.553314 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.553296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:34.556350 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.556321 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:15:34.556350 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.556341 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:15:34.563851 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.563823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4"] Jul 29 16:15:34.648492 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.648449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4\" (UID: \"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:34.749336 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.749299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4\" (UID: \"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:34.749925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.749903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4\" (UID: \"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:34.865706 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:34.865625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:35.018626 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:35.018600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4"] Jul 29 16:15:35.019601 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:15:35.019573 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42ee60a_4f44_4ba6_80d1_4f3f6e4a4d02.slice/crio-e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120 WatchSource:0}: Error finding container e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120: Status 404 returned error can't find the container with id e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120 Jul 29 16:15:35.520339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:35.520305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" event={"ID":"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02","Type":"ContainerStarted","Data":"28b8be9ce6fc9ba384c6f41d2eb64ca22fba9b579b900d3181ba62f7ff3ab499"} Jul 29 16:15:35.520532 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:35.520369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" event={"ID":"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02","Type":"ContainerStarted","Data":"e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120"} Jul 29 16:15:35.538315 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:35.538258 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" podStartSLOduration=1.538239172 podStartE2EDuration="1.538239172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:15:35.536724728 +0000 UTC m=+1731.775281937" watchObservedRunningTime="2026-07-29 16:15:35.538239172 +0000 UTC m=+1731.776796382" Jul 29 16:15:36.527273 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:36.527235 2574 generic.go:356] "Generic (PLEG): container finished" podID="b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" containerID="28b8be9ce6fc9ba384c6f41d2eb64ca22fba9b579b900d3181ba62f7ff3ab499" exitCode=0 Jul 29 16:15:36.527673 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:36.527288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" event={"ID":"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02","Type":"ContainerDied","Data":"28b8be9ce6fc9ba384c6f41d2eb64ca22fba9b579b900d3181ba62f7ff3ab499"} Jul 29 16:15:37.670990 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:37.670950 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:37.778849 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:37.778810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca\") pod \"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02\" (UID: \"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02\") " Jul 29 16:15:37.779230 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:37.779206 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" (UID: "b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:37.880068 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:37.879957 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:38.539460 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:38.539424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" event={"ID":"b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02","Type":"ContainerDied","Data":"e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120"} Jul 29 16:15:38.539460 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:38.539453 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4" Jul 29 16:15:38.539682 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:38.539483 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63c7c94c15af9304aafa55c56d81c210aedd417b71510604bfbc9f8faa71120" Jul 29 16:15:40.476331 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:40.476293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:40.572561 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:40.572528 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:40.725027 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:40.724948 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4"] Jul 29 16:15:40.727606 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:40.727526 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d49b3458-cl7r4"] Jul 29 16:15:41.066118 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:41.066027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:41.156730 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:41.156698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:41.862318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:41.862286 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:41.862795 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:41.862566 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" containerID="cri-o://0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b" gracePeriod=30 Jul 29 16:15:41.880780 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:41.880749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:42.188294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.188207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:42.333791 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.333752 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" path="/var/lib/kubelet/pods/b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02/volumes" Jul 29 16:15:42.390545 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.390513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:42.736037 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736005 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:15:42.736399 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:42.736451 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736401 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:42.736451 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736408 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" containerName="revoke-keys" Jul 29 16:15:42.736451 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736414 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" containerName="revoke-keys" Jul 29 16:15:42.736538 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736476 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb5c544-a384-460a-9302-a16d6b6e442a" containerName="istio-proxy" Jul 29 16:15:42.736538 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.736488 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42ee60a-4f44-4ba6-80d1-4f3f6e4a4d02" containerName="revoke-keys" Jul 29 16:15:42.738728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.738708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.745797 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.745776 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-facc8562-openshift-default-dockercfg-2glzq\"" Jul 29 16:15:42.759494 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.759462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:15:42.785378 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.785346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:42.828562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828761 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828761 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pbl\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828761 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.828929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.829170 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.828952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930215 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930215 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.930699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931045 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pbl\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931045 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931045 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931045 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931045 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.930965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.931283 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.931253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.932608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.932584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.932908 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.932891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.940709 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.940685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:42.941250 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:42.941232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pbl\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl\") pod \"e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:43.049507 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.049421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:43.144069 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.144027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:43.200051 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.200024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:15:43.202027 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:15:43.201998 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c40daee_845e_4ec1_882d_ddc35a8b13ce.slice/crio-35e80a264aa5d2d744abe7efe6df1c4e1a0e8bf961760d9d45de8a7a6a9cddcb WatchSource:0}: Error finding container 35e80a264aa5d2d744abe7efe6df1c4e1a0e8bf961760d9d45de8a7a6a9cddcb: Status 404 returned error can't find the container with id 35e80a264aa5d2d744abe7efe6df1c4e1a0e8bf961760d9d45de8a7a6a9cddcb Jul 29 16:15:43.204127 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.204099 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:43.204181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.204166 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:43.204229 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.204217 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:43.567786 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.567752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerStarted","Data":"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5"} Jul 29 16:15:43.568030 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.567807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerStarted","Data":"35e80a264aa5d2d744abe7efe6df1c4e1a0e8bf961760d9d45de8a7a6a9cddcb"} Jul 29 16:15:43.595961 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:43.595899 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podStartSLOduration=1.595878527 podStartE2EDuration="1.595878527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:15:43.593387245 +0000 UTC m=+1739.831944491" watchObservedRunningTime="2026-07-29 16:15:43.595878527 +0000 UTC m=+1739.834435720" Jul 29 16:15:44.050538 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.050494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:44.055408 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.055382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:44.573165 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.573129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:44.574202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.574182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:15:44.925503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.925460 2574 patch_prober.go:28] interesting pod/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:15:44.925685 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.925551 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:15:44.973472 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:44.973439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:45.263540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:45.263503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:45.368184 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:45.368155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:45.490744 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:45.490708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:45.732670 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:45.732636 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:45.823371 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:45.823337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:15:47.122786 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.122758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:47.173948 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.173860 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.173948 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.173911 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m9kf\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.173953 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174018 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174059 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174079 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174142 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174202 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174173 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174539 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174204 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket\") pod \"a11e6bd9-38fc-44de-b873-8c5b432831b9\" (UID: \"a11e6bd9-38fc-44de-b873-8c5b432831b9\") " Jul 29 16:15:47.174539 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:47.174539 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174457 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:15:47.174690 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174562 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11e6bd9-38fc-44de-b873-8c5b432831b9-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.174690 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174580 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.174690 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174673 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:47.174847 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.174695 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:47.175339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.175314 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:47.176313 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.176285 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:15:47.176647 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.176628 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:15:47.176720 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.176690 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf" (OuterVolumeSpecName: "kube-api-access-7m9kf") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "kube-api-access-7m9kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:47.176820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.176795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a11e6bd9-38fc-44de-b873-8c5b432831b9" (UID: "a11e6bd9-38fc-44de-b873-8c5b432831b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:15:47.275789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275755 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9kf\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-kube-api-access-7m9kf\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.275789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275784 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.275789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275795 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.276056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275804 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.276056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275812 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.276056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275820 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11e6bd9-38fc-44de-b873-8c5b432831b9-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.276056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.275827 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11e6bd9-38fc-44de-b873-8c5b432831b9-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:15:47.591375 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.591339 2574 generic.go:356] "Generic (PLEG): container finished" podID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerID="0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b" exitCode=0 Jul 29 16:15:47.591586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.591405 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" Jul 29 16:15:47.591586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.591413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerDied","Data":"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b"} Jul 29 16:15:47.591586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.591476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98" event={"ID":"a11e6bd9-38fc-44de-b873-8c5b432831b9","Type":"ContainerDied","Data":"4746d4dab57df6245c04adeb739a2a6541c7c74e83446da89a6257b0c7c271c7"} Jul 29 16:15:47.591586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.591499 2574 scope.go:117] "RemoveContainer" containerID="0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b" Jul 29 16:15:47.608457 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.608428 2574 scope.go:117] "RemoveContainer" containerID="92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913" Jul 29 16:15:47.619167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.619140 2574 scope.go:117] "RemoveContainer" containerID="0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b" Jul 29 16:15:47.619494 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:15:47.619471 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b\": container with ID starting with 0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b not found: ID does not exist" containerID="0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b" Jul 29 16:15:47.619567 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.619503 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b"} err="failed to get container status \"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b\": rpc error: code = NotFound desc = could not find container \"0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b\": container with ID starting with 0d6b3f773028329997a35ca05d1cd6d3282724b18507a9dae49c93c3b239aa1b not found: ID does not exist" Jul 29 16:15:47.619567 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.619522 2574 scope.go:117] "RemoveContainer" containerID="92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913" Jul 29 16:15:47.619808 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:15:47.619791 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913\": container with ID starting with 92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913 not found: ID does not exist" containerID="92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913" Jul 29 16:15:47.619854 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.619814 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913"} err="failed to get container status \"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913\": rpc error: code = NotFound desc = could not find container \"92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913\": container with ID starting with 92210851c92de89409bdd4707472bbde2c8485e6e01431a07b0c69fade8ea913 not found: ID does not exist" Jul 29 16:15:47.646517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.646484 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:47.650929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:47.650898 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d49b3458-openshift-default-7685497b6-f8v98"] Jul 29 16:15:48.333783 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:48.333751 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" path="/var/lib/kubelet/pods/a11e6bd9-38fc-44de-b873-8c5b432831b9/volumes" Jul 29 16:15:59.574243 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.574209 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 29 16:15:59.574638 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.574273 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 29 16:15:59.655921 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.655893 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerID="bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5" exitCode=0 Jul 29 16:15:59.656093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.655970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerDied","Data":"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5"} Jul 29 16:15:59.656408 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.656393 2574 scope.go:117] "RemoveContainer" containerID="bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5" Jul 29 16:15:59.657469 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.657434 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:59.657565 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.657499 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:15:59.657565 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:15:59.657538 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:16:00.663345 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:00.663309 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerStarted","Data":"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b"} Jul 29 16:16:01.050112 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:01.050071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:01.050456 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:01.050433 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 29 16:16:01.050520 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:01.050497 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 29 16:16:02.050584 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:02.050551 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 29 16:16:02.051023 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:02.050603 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 29 16:16:03.050639 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:03.050605 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 29 16:16:03.051066 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:03.050664 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 29 16:16:04.050357 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:04.050323 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 29 16:16:04.050545 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:04.050394 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 29 16:16:04.304608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:04.304509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:04.378805 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:04.378774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:05.054090 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:05.054062 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:05.054365 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:05.054346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:05.055080 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:05.055062 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:09.825950 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.825920 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm"] Jul 29 16:16:09.826402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826335 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.826402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826347 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.826402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826360 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.826402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826366 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.826533 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826426 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.826533 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.826436 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11e6bd9-38fc-44de-b873-8c5b432831b9" containerName="istio-proxy" Jul 29 16:16:09.828575 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.828560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:09.831297 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.831271 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:16:09.831443 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.831375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:16:09.837994 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.837958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm"] Jul 29 16:16:09.883105 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.883075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-facc8562-rhtfm\" (UID: \"55296a10-0baf-47ff-bc87-64c4393e13cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:09.984251 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.984221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-facc8562-rhtfm\" (UID: \"55296a10-0baf-47ff-bc87-64c4393e13cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:09.984832 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:09.984812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-facc8562-rhtfm\" (UID: \"55296a10-0baf-47ff-bc87-64c4393e13cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:10.140059 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:10.139958 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:10.275102 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:10.275058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm"] Jul 29 16:16:10.277304 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:16:10.277269 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55296a10_0baf_47ff_bc87_64c4393e13cc.slice/crio-ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931 WatchSource:0}: Error finding container ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931: Status 404 returned error can't find the container with id ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931 Jul 29 16:16:10.713621 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:10.713578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" event={"ID":"55296a10-0baf-47ff-bc87-64c4393e13cc","Type":"ContainerStarted","Data":"804f5117d712bbf6b84584656947082758983648e4e7ef24c63321d3f2561dc1"} Jul 29 16:16:10.713823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:10.713669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" event={"ID":"55296a10-0baf-47ff-bc87-64c4393e13cc","Type":"ContainerStarted","Data":"ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931"} Jul 29 16:16:10.730562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:10.730516 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" podStartSLOduration=1.730503595 podStartE2EDuration="1.730503595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:16:10.730260958 +0000 UTC m=+1766.968818171" watchObservedRunningTime="2026-07-29 16:16:10.730503595 +0000 UTC m=+1766.969060806" Jul 29 16:16:11.720517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:11.720488 2574 generic.go:356] "Generic (PLEG): container finished" podID="55296a10-0baf-47ff-bc87-64c4393e13cc" containerID="804f5117d712bbf6b84584656947082758983648e4e7ef24c63321d3f2561dc1" exitCode=0 Jul 29 16:16:11.720892 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:11.720570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" event={"ID":"55296a10-0baf-47ff-bc87-64c4393e13cc","Type":"ContainerDied","Data":"804f5117d712bbf6b84584656947082758983648e4e7ef24c63321d3f2561dc1"} Jul 29 16:16:12.860491 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:12.860469 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:12.912469 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:12.912437 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca\") pod \"55296a10-0baf-47ff-bc87-64c4393e13cc\" (UID: \"55296a10-0baf-47ff-bc87-64c4393e13cc\") " Jul 29 16:16:12.912789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:12.912766 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55296a10-0baf-47ff-bc87-64c4393e13cc" (UID: "55296a10-0baf-47ff-bc87-64c4393e13cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:16:13.013856 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:13.013776 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55296a10-0baf-47ff-bc87-64c4393e13cc-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:13.732570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:13.732531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" event={"ID":"55296a10-0baf-47ff-bc87-64c4393e13cc","Type":"ContainerDied","Data":"ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931"} Jul 29 16:16:13.732570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:13.732549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm" Jul 29 16:16:13.732776 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:13.732582 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8bb837bc4ba6273aecbe817d65b6853905d03feade4cdc09426700ebe7d931" Jul 29 16:16:16.007822 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.007788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:16.087198 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.087165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:16.186967 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.186931 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm"] Jul 29 16:16:16.188689 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.188666 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-facc8562-rhtfm"] Jul 29 16:16:16.189630 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.189606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:16.256481 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.256451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:16.331552 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.325291 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:16:16.331552 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.325615 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" containerID="cri-o://8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b" gracePeriod=30 Jul 29 16:16:16.337206 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.337177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:16.341442 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.341415 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55296a10-0baf-47ff-bc87-64c4393e13cc" path="/var/lib/kubelet/pods/55296a10-0baf-47ff-bc87-64c4393e13cc/volumes" Jul 29 16:16:16.598144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:16.598062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:17.074678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:17.074636 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:17.257626 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:17.257590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:17.496165 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:17.496129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:18.813053 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:18.813022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:20.056603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:20.056560 2574 patch_prober.go:28] interesting pod/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:16:20.056952 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:20.056624 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:16:21.644902 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.644877 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:21.694576 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694542 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694597 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42pbl\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694693 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.694766 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694761 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.695167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694817 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.695167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694847 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data\") pod \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\" (UID: \"6c40daee-845e-4ec1-882d-ddc35a8b13ce\") " Jul 29 16:16:21.695167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.694910 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:16:21.695167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695096 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:16:21.695380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695238 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.695380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695260 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.695380 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695305 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:16:21.695868 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695845 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:16:21.696031 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.695865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:16:21.697572 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.697527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:16:21.697697 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.697599 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:16:21.697787 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.697707 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl" (OuterVolumeSpecName: "kube-api-access-42pbl") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "kube-api-access-42pbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:16:21.698151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.698129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c40daee-845e-4ec1-882d-ddc35a8b13ce" (UID: "6c40daee-845e-4ec1-882d-ddc35a8b13ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:16:21.776067 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.776032 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerID="8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b" exitCode=0 Jul 29 16:16:21.776239 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.776095 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" Jul 29 16:16:21.776239 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.776100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerDied","Data":"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b"} Jul 29 16:16:21.776239 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.776234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh" event={"ID":"6c40daee-845e-4ec1-882d-ddc35a8b13ce","Type":"ContainerDied","Data":"35e80a264aa5d2d744abe7efe6df1c4e1a0e8bf961760d9d45de8a7a6a9cddcb"} Jul 29 16:16:21.776401 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.776263 2574 scope.go:117] "RemoveContainer" containerID="8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b" Jul 29 16:16:21.786503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.786469 2574 scope.go:117] "RemoveContainer" containerID="bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5" Jul 29 16:16:21.795359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795334 2574 scope.go:117] "RemoveContainer" containerID="8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b" Jul 29 16:16:21.795682 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795662 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795682 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:16:21.795672 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b\": container with ID starting with 8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b not found: ID does not exist" containerID="8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795685 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pbl\" (UniqueName: \"kubernetes.io/projected/6c40daee-845e-4ec1-882d-ddc35a8b13ce-kube-api-access-42pbl\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795695 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795705 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795701 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b"} err="failed to get container status \"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b\": rpc error: code = NotFound desc = could not find container \"8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b\": container with ID starting with 8f7126c2e2787432f92362341c5adf83cca46b49cdd3a52ee079458ca9587b9b not found: ID does not exist" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795713 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795721 2574 scope.go:117] "RemoveContainer" containerID="bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795722 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.795793 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.795786 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c40daee-845e-4ec1-882d-ddc35a8b13ce-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:16:21.796139 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:16:21.796007 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5\": container with ID starting with bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5 not found: ID does not exist" containerID="bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5" Jul 29 16:16:21.796139 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.796029 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5"} err="failed to get container status \"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5\": rpc error: code = NotFound desc = could not find container \"bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5\": container with ID starting with bf2408f9fe49c3672e2c8153fee22657cf6f55f686cf3f3ce4e8df5eae3469d5 not found: ID does not exist" Jul 29 16:16:21.800365 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.800340 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:16:21.808873 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:21.808849 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-facc8562-openshift-default-7bc487cb8-2kzbh"] Jul 29 16:16:22.335636 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:22.335602 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" path="/var/lib/kubelet/pods/6c40daee-845e-4ec1-882d-ddc35a8b13ce/volumes" Jul 29 16:16:24.695073 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:24.695040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:24.777106 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:24.777069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:24.851493 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:24.851462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:24.955883 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:24.955851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:25.481333 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:25.481302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:44.428821 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:44.428795 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:16:44.442668 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:44.442643 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:16:51.388627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:51.388597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:57.486339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:57.486288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:57.592229 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:57.592193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:16:57.685005 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:16:57.684950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:13.548837 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:13.548796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:19.418193 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:19.418161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:19.479083 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:19.479047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:19.574199 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:19.574168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:19.674501 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:19.674407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:27.902962 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:27.902920 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:28.003070 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:28.003039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:30.133867 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:30.133821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:30.923328 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:30.923296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:38.475490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:38.475446 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:38.594584 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:38.594548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:38.741956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:38.741864 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:38.846259 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:38.846218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:51.280478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:51.280439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:51.494867 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:51.494838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:53.361188 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:53.361155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:59.443294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:59.443260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:17:59.544928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:17:59.544895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:00.147044 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:00.147002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:06.375822 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.375737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:06.478368 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.478337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:06.616065 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.616027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:06.926274 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926230 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:18:06.926688 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926670 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.926778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926690 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.926778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926704 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.926778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926711 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.926778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926732 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55296a10-0baf-47ff-bc87-64c4393e13cc" containerName="revoke-keys" Jul 29 16:18:06.926778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926741 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="55296a10-0baf-47ff-bc87-64c4393e13cc" containerName="revoke-keys" Jul 29 16:18:06.927054 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926846 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.927054 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926858 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="55296a10-0baf-47ff-bc87-64c4393e13cc" containerName="revoke-keys" Jul 29 16:18:06.927054 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.926869 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c40daee-845e-4ec1-882d-ddc35a8b13ce" containerName="istio-proxy" Jul 29 16:18:06.930413 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.930394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:06.933736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.933718 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-82c422a4-openshift-default-dockercfg-tdphc\"" Jul 29 16:18:06.961654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.961630 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:18:06.969331 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:06.969305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:07.070446 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz849\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070806 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.070806 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.070685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.171856 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.171824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.171856 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.171861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz849\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172075 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.171880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172075 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.171907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172075 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.171958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172075 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172520 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.172851 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.172830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.174331 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.174312 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.174442 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.174423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.183890 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.183833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.184720 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.184701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz849\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849\") pod \"e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.240502 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.240472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:07.390643 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.390619 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:18:07.392521 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:18:07.392489 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bbdf55_7ef6_4ca9_84bf_bc30e8d24428.slice/crio-2e287dee3f4c40bc9228ec8197ea96b491b916fe6c6247a5045117e2558a7898 WatchSource:0}: Error finding container 2e287dee3f4c40bc9228ec8197ea96b491b916fe6c6247a5045117e2558a7898: Status 404 returned error can't find the container with id 2e287dee3f4c40bc9228ec8197ea96b491b916fe6c6247a5045117e2558a7898 Jul 29 16:18:07.395039 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.395006 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:07.395156 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.395084 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:07.395156 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.395128 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:07.536919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:07.536885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:08.311931 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:08.311899 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerStarted","Data":"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a"} Jul 29 16:18:08.312112 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:08.311947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerStarted","Data":"2e287dee3f4c40bc9228ec8197ea96b491b916fe6c6247a5045117e2558a7898"} Jul 29 16:18:08.338042 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:08.337998 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podStartSLOduration=2.33796745 podStartE2EDuration="2.33796745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:18:08.336217885 +0000 UTC m=+1884.574775095" watchObservedRunningTime="2026-07-29 16:18:08.33796745 +0000 UTC m=+1884.576524660" Jul 29 16:18:09.241609 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:09.241582 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:09.246919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:09.246897 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:09.316946 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:09.316917 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:09.317783 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:09.317768 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:10.274383 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:10.274331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:10.374747 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:10.374717 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:10.690857 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:10.690759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:11.291625 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:11.291559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:11.760104 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:11.760066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:11.913190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:11.913155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:24.805353 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:24.805310 2574 patch_prober.go:28] interesting pod/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:37958->10.134.0.51:15021: read: connection reset by peer" start-of-body= Jul 29 16:18:24.805759 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:24.805404 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:37958->10.134.0.51:15021: read: connection reset by peer" Jul 29 16:18:25.407519 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.407486 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerID="4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a" exitCode=0 Jul 29 16:18:25.407698 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.407547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerDied","Data":"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a"} Jul 29 16:18:25.407930 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.407914 2574 scope.go:117] "RemoveContainer" containerID="4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a" Jul 29 16:18:25.408973 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.408936 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:25.409076 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.409044 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:25.409138 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:25.409085 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:18:26.414364 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:26.414322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerStarted","Data":"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f"} Jul 29 16:18:27.240784 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:27.240744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:27.241062 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:27.241038 2574 patch_prober.go:28] interesting pod/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 29 16:18:27.241160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:27.241096 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 29 16:18:28.241799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:28.241760 2574 patch_prober.go:28] interesting pod/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 29 16:18:28.242222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:28.241816 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 29 16:18:29.241098 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:29.241066 2574 patch_prober.go:28] interesting pod/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 29 16:18:29.241266 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:29.241117 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 29 16:18:30.241906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:30.241864 2574 patch_prober.go:28] interesting pod/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 29 16:18:30.242423 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:30.241939 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 29 16:18:31.244728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:31.244695 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:31.245125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:31.244950 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:31.245680 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:31.245662 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:18:35.238053 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:35.238017 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:35.887384 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:35.887340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:35.982236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:35.982189 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:36.321330 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:36.321266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:43.124005 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.123942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:43.542880 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.542843 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn"] Jul 29 16:18:43.545574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.545554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:43.548886 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.548867 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:18:43.549274 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.549256 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:18:43.563396 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.563370 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn"] Jul 29 16:18:43.627646 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.627613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82c422a4-8njjn\" (UID: \"e60d5eae-c777-4763-a261-a41de26f5d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:43.668479 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.668450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:43.728852 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.728819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82c422a4-8njjn\" (UID: \"e60d5eae-c777-4763-a261-a41de26f5d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:43.729461 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.729444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82c422a4-8njjn\" (UID: \"e60d5eae-c777-4763-a261-a41de26f5d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:43.856191 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:43.856115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:44.003066 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:44.003041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn"] Jul 29 16:18:44.005451 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:18:44.005425 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60d5eae_c777_4763_a261_a41de26f5d66.slice/crio-0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3 WatchSource:0}: Error finding container 0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3: Status 404 returned error can't find the container with id 0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3 Jul 29 16:18:44.507048 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:44.507012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" event={"ID":"e60d5eae-c777-4763-a261-a41de26f5d66","Type":"ContainerStarted","Data":"4c91c77bac7ad33fb0105502eecc1d4fedf1936244b60cc4f5dfb7c9cfd6639f"} Jul 29 16:18:44.507434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:44.507076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" event={"ID":"e60d5eae-c777-4763-a261-a41de26f5d66","Type":"ContainerStarted","Data":"0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3"} Jul 29 16:18:44.540430 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:44.540374 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" podStartSLOduration=1.5403584989999999 podStartE2EDuration="1.540358499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:18:44.539200728 +0000 UTC m=+1920.777757938" watchObservedRunningTime="2026-07-29 16:18:44.540358499 +0000 UTC m=+1920.778915708" Jul 29 16:18:45.513232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:45.513197 2574 generic.go:356] "Generic (PLEG): container finished" podID="e60d5eae-c777-4763-a261-a41de26f5d66" containerID="4c91c77bac7ad33fb0105502eecc1d4fedf1936244b60cc4f5dfb7c9cfd6639f" exitCode=0 Jul 29 16:18:45.513606 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:45.513255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" event={"ID":"e60d5eae-c777-4763-a261-a41de26f5d66","Type":"ContainerDied","Data":"4c91c77bac7ad33fb0105502eecc1d4fedf1936244b60cc4f5dfb7c9cfd6639f"} Jul 29 16:18:46.648389 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:46.648364 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:46.758434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:46.758406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca\") pod \"e60d5eae-c777-4763-a261-a41de26f5d66\" (UID: \"e60d5eae-c777-4763-a261-a41de26f5d66\") " Jul 29 16:18:46.758751 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:46.758728 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e60d5eae-c777-4763-a261-a41de26f5d66" (UID: "e60d5eae-c777-4763-a261-a41de26f5d66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:18:46.859145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:46.859065 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60d5eae-c777-4763-a261-a41de26f5d66-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:18:47.525712 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:47.525685 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" Jul 29 16:18:47.525885 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:47.525709 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn" event={"ID":"e60d5eae-c777-4763-a261-a41de26f5d66","Type":"ContainerDied","Data":"0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3"} Jul 29 16:18:47.525885 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:47.525757 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0131d64368866cc93def13a31017bf1e6cda292aaf78ec2d02a1211153a8bfc3" Jul 29 16:18:54.310104 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.310066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:54.445293 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.445263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:54.535311 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.535274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:54.671163 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.671084 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn"] Jul 29 16:18:54.676101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.676077 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82c422a4-8njjn"] Jul 29 16:18:54.794203 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:54.794162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:55.099093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:55.099044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:18:56.333465 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:18:56.333432 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60d5eae-c777-4763-a261-a41de26f5d66" path="/var/lib/kubelet/pods/e60d5eae-c777-4763-a261-a41de26f5d66/volumes" Jul 29 16:19:01.454889 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:01.454852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:01.554998 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:01.554950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:01.704583 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:01.704547 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:19:01.704805 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:01.704778 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" containerID="cri-o://e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f" gracePeriod=30 Jul 29 16:19:02.033259 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:02.033175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:02.212799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:02.212764 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:02.321973 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:02.321894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:06.960173 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:06.960145 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:19:07.044654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044615 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044679 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz849\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044743 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044833 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044843 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044861 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.044928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.044901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token\") pod \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\" (UID: \"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428\") " Jul 29 16:19:07.045311 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045065 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:19:07.045311 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:19:07.045311 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045223 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:19:07.045471 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045423 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.045471 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045444 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.045471 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045457 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.045633 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045470 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.045836 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.045813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:19:07.046832 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.046807 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:19:07.047111 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.047094 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:19:07.047251 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.047236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:19:07.047296 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.047264 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849" (OuterVolumeSpecName: "kube-api-access-cz849") pod "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" (UID: "e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428"). InnerVolumeSpecName "kube-api-access-cz849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:19:07.146101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.146065 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.146101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.146094 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.146101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.146102 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.146101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.146111 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.146369 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.146119 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz849\" (UniqueName: \"kubernetes.io/projected/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428-kube-api-access-cz849\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:19:07.628365 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.628329 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerID="e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f" exitCode=0 Jul 29 16:19:07.628534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.628378 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" Jul 29 16:19:07.628534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.628417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerDied","Data":"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f"} Jul 29 16:19:07.628534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.628471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6" event={"ID":"e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428","Type":"ContainerDied","Data":"2e287dee3f4c40bc9228ec8197ea96b491b916fe6c6247a5045117e2558a7898"} Jul 29 16:19:07.628534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.628494 2574 scope.go:117] "RemoveContainer" containerID="e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f" Jul 29 16:19:07.637792 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.637777 2574 scope.go:117] "RemoveContainer" containerID="4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a" Jul 29 16:19:07.645382 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.645368 2574 scope.go:117] "RemoveContainer" containerID="e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f" Jul 29 16:19:07.645635 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:19:07.645612 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f\": container with ID starting with e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f not found: ID does not exist" containerID="e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f" Jul 29 16:19:07.645688 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.645646 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f"} err="failed to get container status \"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f\": rpc error: code = NotFound desc = could not find container \"e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f\": container with ID starting with e46a280c9de17642c2f6a1a5e9ff58d6a35369fe55fa6eaffb5716ac195d606f not found: ID does not exist" Jul 29 16:19:07.645688 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.645667 2574 scope.go:117] "RemoveContainer" containerID="4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a" Jul 29 16:19:07.645875 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:19:07.645859 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a\": container with ID starting with 4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a not found: ID does not exist" containerID="4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a" Jul 29 16:19:07.645919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.645880 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a"} err="failed to get container status \"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a\": rpc error: code = NotFound desc = could not find container \"4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a\": container with ID starting with 4a49ba30a56e9994b8bd4797dcec9eb618bcc54e7c0f0e88f34caff4d781663a not found: ID does not exist" Jul 29 16:19:07.657402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.657375 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:19:07.665891 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:07.665871 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-82c422a4-openshift-default-57d5d4c45b-zv8k6"] Jul 29 16:19:08.333508 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:08.333475 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" path="/var/lib/kubelet/pods/e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428/volumes" Jul 29 16:19:09.564396 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:09.564359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:10.147885 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:10.147846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:10.264552 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:10.264520 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:15.848906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:15.848876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:16.400178 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:16.400143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:18.051092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:18.051057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:24.871172 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:24.871136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:25.167071 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.166480 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167318 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60d5eae-c777-4763-a261-a41de26f5d66" containerName="revoke-keys" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167339 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60d5eae-c777-4763-a261-a41de26f5d66" containerName="revoke-keys" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167359 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167368 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167560 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60d5eae-c777-4763-a261-a41de26f5d66" containerName="revoke-keys" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167572 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:19:25.169286 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.167581 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:19:25.170676 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.170652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.173356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.173336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6a1f06a-openshift-default-dockercfg-9mvx9\"" Jul 29 16:19:25.183319 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.183293 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:19:25.199926 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.199893 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:25.303087 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgkn\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.303503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.303324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.403988 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.403954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404138 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404219 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404219 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404219 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404535 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgkn\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404535 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404650 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404650 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.404817 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.404794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.405147 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.405127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.406550 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.406529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.406670 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.406650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.413426 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.413402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.413580 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.413563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgkn\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn\") pod \"e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.480774 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.480743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:25.572155 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.572126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:25.621770 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.621746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:19:25.623858 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:19:25.623807 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660163a5_c6f4_47aa_9021_3371c9530a2b.slice/crio-48b1f796b20f31eced12bf309392aa2dc6f5cf59c55dbb6ddb5b7012c83b70c3 WatchSource:0}: Error finding container 48b1f796b20f31eced12bf309392aa2dc6f5cf59c55dbb6ddb5b7012c83b70c3: Status 404 returned error can't find the container with id 48b1f796b20f31eced12bf309392aa2dc6f5cf59c55dbb6ddb5b7012c83b70c3 Jul 29 16:19:25.626745 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.626702 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:25.626853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.626787 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:25.626853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.626826 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:25.721562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.721524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerStarted","Data":"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22"} Jul 29 16:19:25.721754 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.721591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerStarted","Data":"48b1f796b20f31eced12bf309392aa2dc6f5cf59c55dbb6ddb5b7012c83b70c3"} Jul 29 16:19:25.747948 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:25.747861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podStartSLOduration=0.747847052 podStartE2EDuration="747.847052ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:19:25.745883526 +0000 UTC m=+1961.984440942" watchObservedRunningTime="2026-07-29 16:19:25.747847052 +0000 UTC m=+1961.986404261" Jul 29 16:19:26.481608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:26.481579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:26.486291 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:26.486268 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:26.726949 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:26.726922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:26.727730 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:26.727710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:27.674213 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:27.674185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:27.781029 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:27.780972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:27.874996 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:27.874960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:28.041892 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:28.041855 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:28.586740 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:28.586705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:28.683839 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:28.683804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:19:42.319708 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.319081 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:58148->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 29 16:19:42.319708 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.319157 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:58148->10.134.0.53:15021: read: connection reset by peer" Jul 29 16:19:42.811707 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.811670 2574 generic.go:356] "Generic (PLEG): container finished" podID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerID="46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22" exitCode=0 Jul 29 16:19:42.811899 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.811746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerDied","Data":"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22"} Jul 29 16:19:42.812342 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.812323 2574 scope.go:117] "RemoveContainer" containerID="46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22" Jul 29 16:19:42.813394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.813366 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:42.813486 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.813433 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:42.813486 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:42.813465 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:19:43.822156 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:43.822120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerStarted","Data":"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5"} Jul 29 16:19:44.481317 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:44.481295 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:44.481537 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:44.481516 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 16:19:44.481603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:44.481587 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 16:19:45.481447 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:45.481408 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 16:19:45.481835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:45.481480 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 16:19:46.482040 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:46.482003 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 16:19:46.482433 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:46.482059 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 16:19:47.481379 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:47.481337 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 29 16:19:47.481556 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:47.481400 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 29 16:19:48.484960 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:48.484931 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:48.485408 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:48.485243 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:48.485868 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:48.485854 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:19:52.636847 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:19:52.636814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:02.385489 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:02.385446 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:11.642844 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:11.642806 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:32.030835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.030790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8"] Jul 29 16:20:32.031504 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.031482 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:20:32.031549 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.031509 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9bbdf55-7ef6-4ca9-84bf-bc30e8d24428" containerName="istio-proxy" Jul 29 16:20:32.034245 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.034222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:32.036758 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.036737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:20:32.036924 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.036902 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:20:32.040724 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.040702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8"] Jul 29 16:20:32.134949 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.134914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8\" (UID: \"0e10bbbe-bfac-458a-848d-d8fe17b5436e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:32.236191 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.236150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8\" (UID: \"0e10bbbe-bfac-458a-848d-d8fe17b5436e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:32.236772 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.236752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8\" (UID: \"0e10bbbe-bfac-458a-848d-d8fe17b5436e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:32.345656 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.345570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:32.481387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:32.481359 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8"] Jul 29 16:20:32.483637 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:20:32.483610 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e10bbbe_bfac_458a_848d_d8fe17b5436e.slice/crio-6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d WatchSource:0}: Error finding container 6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d: Status 404 returned error can't find the container with id 6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d Jul 29 16:20:33.086660 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:33.086626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" event={"ID":"0e10bbbe-bfac-458a-848d-d8fe17b5436e","Type":"ContainerStarted","Data":"0962c69225acc689652db9e8f4519c13e3ab69e06033880eb857ecf48b22e696"} Jul 29 16:20:33.087103 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:33.086692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" event={"ID":"0e10bbbe-bfac-458a-848d-d8fe17b5436e","Type":"ContainerStarted","Data":"6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d"} Jul 29 16:20:33.104075 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:33.104019 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" podStartSLOduration=1.104002272 podStartE2EDuration="1.104002272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:20:33.10290185 +0000 UTC m=+2029.341459061" watchObservedRunningTime="2026-07-29 16:20:33.104002272 +0000 UTC m=+2029.342559483" Jul 29 16:20:34.094217 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:34.094183 2574 generic.go:356] "Generic (PLEG): container finished" podID="0e10bbbe-bfac-458a-848d-d8fe17b5436e" containerID="0962c69225acc689652db9e8f4519c13e3ab69e06033880eb857ecf48b22e696" exitCode=0 Jul 29 16:20:34.094619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:34.094256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" event={"ID":"0e10bbbe-bfac-458a-848d-d8fe17b5436e","Type":"ContainerDied","Data":"0962c69225acc689652db9e8f4519c13e3ab69e06033880eb857ecf48b22e696"} Jul 29 16:20:35.235577 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:35.235553 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:35.364058 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:35.363948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca\") pod \"0e10bbbe-bfac-458a-848d-d8fe17b5436e\" (UID: \"0e10bbbe-bfac-458a-848d-d8fe17b5436e\") " Jul 29 16:20:35.364290 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:35.364266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e10bbbe-bfac-458a-848d-d8fe17b5436e" (UID: "0e10bbbe-bfac-458a-848d-d8fe17b5436e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:20:35.465827 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:35.465785 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e10bbbe-bfac-458a-848d-d8fe17b5436e-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:20:36.110497 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:36.110461 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" Jul 29 16:20:36.110497 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:36.110479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8" event={"ID":"0e10bbbe-bfac-458a-848d-d8fe17b5436e","Type":"ContainerDied","Data":"6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d"} Jul 29 16:20:36.110767 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:36.110533 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af0e3fe5144ffe40fcd9f8cdda247075e98f68a4315c9c78673128f3f57332d" Jul 29 16:20:47.799852 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.799821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:47.864656 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.864627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:47.928168 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.928130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:47.973037 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.972972 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8"] Jul 29 16:20:47.974910 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.974889 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6a1f06a-5tmm8"] Jul 29 16:20:47.982340 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:47.982313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:48.333289 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:48.333256 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e10bbbe-bfac-458a-848d-d8fe17b5436e" path="/var/lib/kubelet/pods/0e10bbbe-bfac-458a-848d-d8fe17b5436e/volumes" Jul 29 16:20:51.181263 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:51.181231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:20:56.206784 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:20:56.206751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:32.392338 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:32.392304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:33.272120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:33.272072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:44.458484 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:44.458453 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:21:44.476016 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:44.475968 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:21:44.489628 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:44.489600 2574 scope.go:117] "RemoveContainer" containerID="73339136c60a4124cb793d15774c05eecf0e32e1dcb3bd7b273e5c14393c2bf8" Jul 29 16:21:44.500474 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:44.500446 2574 scope.go:117] "RemoveContainer" containerID="28b8be9ce6fc9ba384c6f41d2eb64ca22fba9b579b900d3181ba62f7ff3ab499" Jul 29 16:21:47.454906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.454872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:47.521991 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.521947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:47.718544 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.718460 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:21:47.718715 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.718673 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" containerID="cri-o://5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5" gracePeriod=30 Jul 29 16:21:47.743673 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.743632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:47.963654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:47.963615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:48.487736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:48.487701 2574 patch_prober.go:28] interesting pod/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:21:48.488136 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:48.487770 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:21:49.015604 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:49.015563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:49.115888 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:49.115858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:49.219152 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:49.219119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:49.944434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:49.944402 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:50.042271 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:50.042239 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:50.190808 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:50.190774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:51.521082 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:51.521054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:53.008888 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.008853 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:21:53.160251 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160215 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160269 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160324 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160352 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160370 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160391 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djgkn\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160436 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160464 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160465 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160840 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo\") pod \"660163a5-c6f4-47aa-9021-3371c9530a2b\" (UID: \"660163a5-c6f4-47aa-9021-3371c9530a2b\") " Jul 29 16:21:53.160840 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160529 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:21:53.160840 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.160777 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.161232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.161026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:21:53.161232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.161063 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:21:53.161232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.161107 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:21:53.161232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.161205 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:21:53.162812 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.162786 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:21:53.162913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.162801 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn" (OuterVolumeSpecName: "kube-api-access-djgkn") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "kube-api-access-djgkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:21:53.162913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.162836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:21:53.163310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.163283 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "660163a5-c6f4-47aa-9021-3371c9530a2b" (UID: "660163a5-c6f4-47aa-9021-3371c9530a2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261649 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261681 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261695 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660163a5-c6f4-47aa-9021-3371c9530a2b-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261706 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261718 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgkn\" (UniqueName: \"kubernetes.io/projected/660163a5-c6f4-47aa-9021-3371c9530a2b-kube-api-access-djgkn\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261732 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.261743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261747 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660163a5-c6f4-47aa-9021-3371c9530a2b-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.262157 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.261758 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660163a5-c6f4-47aa-9021-3371c9530a2b-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:21:53.514372 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.514276 2574 generic.go:356] "Generic (PLEG): container finished" podID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerID="5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5" exitCode=0 Jul 29 16:21:53.514372 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.514343 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" Jul 29 16:21:53.514597 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.514370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerDied","Data":"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5"} Jul 29 16:21:53.514597 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.514427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm" event={"ID":"660163a5-c6f4-47aa-9021-3371c9530a2b","Type":"ContainerDied","Data":"48b1f796b20f31eced12bf309392aa2dc6f5cf59c55dbb6ddb5b7012c83b70c3"} Jul 29 16:21:53.514597 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.514459 2574 scope.go:117] "RemoveContainer" containerID="5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5" Jul 29 16:21:53.524616 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.524593 2574 scope.go:117] "RemoveContainer" containerID="46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22" Jul 29 16:21:53.533596 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.533570 2574 scope.go:117] "RemoveContainer" containerID="5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5" Jul 29 16:21:53.533878 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:21:53.533855 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5\": container with ID starting with 5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5 not found: ID does not exist" containerID="5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5" Jul 29 16:21:53.533928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.533888 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5"} err="failed to get container status \"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5\": rpc error: code = NotFound desc = could not find container \"5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5\": container with ID starting with 5fdf209598714d85b3adb651ec388ef23edd5c8a0f6ee95caf00555ddd06d7b5 not found: ID does not exist" Jul 29 16:21:53.533928 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.533905 2574 scope.go:117] "RemoveContainer" containerID="46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22" Jul 29 16:21:53.534218 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:21:53.534201 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22\": container with ID starting with 46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22 not found: ID does not exist" containerID="46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22" Jul 29 16:21:53.534267 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.534222 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22"} err="failed to get container status \"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22\": rpc error: code = NotFound desc = could not find container \"46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22\": container with ID starting with 46799c5725e071b80a4a3c4983f3a03ca1885fa93dcdfe7c184086d5d24acd22 not found: ID does not exist" Jul 29 16:21:53.537698 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.537675 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:21:53.546869 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:53.546844 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6a1f06a-openshift-default-84ffc54456-tv4tm"] Jul 29 16:21:54.333428 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:54.333399 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" path="/var/lib/kubelet/pods/660163a5-c6f4-47aa-9021-3371c9530a2b/volumes" Jul 29 16:21:57.291034 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:57.290969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:57.391111 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:57.391068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:21:57.964071 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:21:57.964035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:03.816479 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:03.816441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:03.919398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:03.919363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:04.015455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:04.015414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:14.505300 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:14.505245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:14.828335 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:14.828244 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:35.778758 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:35.778658 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:42.623966 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:42.623929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:42.688861 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:42.688820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:42.982241 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:42.982208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:43.273759 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.273684 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:22:43.274093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274079 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274094 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274105 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e10bbbe-bfac-458a-848d-d8fe17b5436e" containerName="revoke-keys" Jul 29 16:22:43.274144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274111 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e10bbbe-bfac-458a-848d-d8fe17b5436e" containerName="revoke-keys" Jul 29 16:22:43.274144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274122 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274128 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274304 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274197 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274304 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274206 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="660163a5-c6f4-47aa-9021-3371c9530a2b" containerName="istio-proxy" Jul 29 16:22:43.274304 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.274212 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e10bbbe-bfac-458a-848d-d8fe17b5436e" containerName="revoke-keys" Jul 29 16:22:43.277375 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.277358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.281582 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.281560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b340d6-openshift-default-dockercfg-rqt7w\"" Jul 29 16:22:43.295432 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.295406 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:22:43.316035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.315998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:43.427540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427779 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427779 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427779 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427727 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427779 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d8n\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427951 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427951 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.427951 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.427866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529388 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529388 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d8n\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529749 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529749 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529749 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529765 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.529906 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.529801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.530079 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.530053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.530172 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.530148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.530241 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.530177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.530611 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.530587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.532070 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.532051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.532271 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.532250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.537872 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.537844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:43.538538 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.538514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.538649 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.538626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d8n\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n\") pod \"e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.587668 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.587638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:43.728930 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.728839 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:22:43.731386 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:22:43.731352 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1394028f_9993_4594_b79b_b84a4f895734.slice/crio-74d1c2fc982ed77da5830ab6c86da275f899ab80dae77f534419c073e89f730a WatchSource:0}: Error finding container 74d1c2fc982ed77da5830ab6c86da275f899ab80dae77f534419c073e89f730a: Status 404 returned error can't find the container with id 74d1c2fc982ed77da5830ab6c86da275f899ab80dae77f534419c073e89f730a Jul 29 16:22:43.733487 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.733459 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:22:43.733574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.733520 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:22:43.733574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.733551 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:22:43.776538 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:43.776511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerStarted","Data":"74d1c2fc982ed77da5830ab6c86da275f899ab80dae77f534419c073e89f730a"} Jul 29 16:22:44.524556 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:44.524520 2574 scope.go:117] "RemoveContainer" containerID="804f5117d712bbf6b84584656947082758983648e4e7ef24c63321d3f2561dc1" Jul 29 16:22:44.783364 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:44.783262 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerStarted","Data":"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80"} Jul 29 16:22:44.804091 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:44.804034 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podStartSLOduration=1.804013344 podStartE2EDuration="1.804013344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:22:44.803850572 +0000 UTC m=+2161.042407781" watchObservedRunningTime="2026-07-29 16:22:44.804013344 +0000 UTC m=+2161.042570559" Jul 29 16:22:45.588299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:45.588260 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:45.594052 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:45.594024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:45.793964 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:45.793927 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:45.794931 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:45.794907 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:22:46.811403 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:46.811367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:46.960276 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:46.960247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:47.185197 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.185109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:47.450863 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.450781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:47.659366 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.659323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:47.759655 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.759619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:22:47.790820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.790790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:22:47.794501 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.794473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:47.797287 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.797264 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b340d6\"" Jul 29 16:22:47.797506 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.797266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b340d6-dockercfg-mjtnp\"" Jul 29 16:22:47.807756 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.807728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:22:47.973179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.973149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:47.973603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:47.973358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqt8\" (UniqueName: \"kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.074807 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.074710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqt8\" (UniqueName: \"kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.074807 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.074803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.075520 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.075496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.083420 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.083397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqt8\" (UniqueName: \"kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8\") pod \"payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.109502 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.109469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.248657 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.248527 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:22:48.251588 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:22:48.251561 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1502de_de66_4d4e_9341_ad12b643468e.slice/crio-eb164557829b1d6f797f4282f20bfe653963b5f85c18945219b5ff5482457656 WatchSource:0}: Error finding container eb164557829b1d6f797f4282f20bfe653963b5f85c18945219b5ff5482457656: Status 404 returned error can't find the container with id eb164557829b1d6f797f4282f20bfe653963b5f85c18945219b5ff5482457656 Jul 29 16:22:48.816350 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.816315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" event={"ID":"df1502de-de66-4d4e-9341-ad12b643468e","Type":"ContainerStarted","Data":"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e"} Jul 29 16:22:48.816554 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.816379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" event={"ID":"df1502de-de66-4d4e-9341-ad12b643468e","Type":"ContainerStarted","Data":"eb164557829b1d6f797f4282f20bfe653963b5f85c18945219b5ff5482457656"} Jul 29 16:22:48.816554 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.816480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:22:48.835291 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:48.835242 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" podStartSLOduration=1.8352274739999999 podStartE2EDuration="1.835227474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:22:48.83429968 +0000 UTC m=+2165.072856894" watchObservedRunningTime="2026-07-29 16:22:48.835227474 +0000 UTC m=+2165.073784685" Jul 29 16:22:59.824505 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:22:59.824473 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:23:00.853907 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.853726 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:43654->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 29 16:23:00.853907 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.853800 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:43654->10.134.0.55:15021: read: connection reset by peer" Jul 29 16:23:00.879238 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.879206 2574 generic.go:356] "Generic (PLEG): container finished" podID="1394028f-9993-4594-b79b-b84a4f895734" containerID="d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80" exitCode=0 Jul 29 16:23:00.879350 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.879273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerDied","Data":"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80"} Jul 29 16:23:00.879673 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.879660 2574 scope.go:117] "RemoveContainer" containerID="d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80" Jul 29 16:23:00.880648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.880626 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:00.880705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.880692 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:00.880742 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:00.880726 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:01.886885 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:01.886849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerStarted","Data":"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d"} Jul 29 16:23:02.588773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:02.588734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:23:02.589040 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:02.589016 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 29 16:23:02.589103 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:02.589077 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 29 16:23:03.588254 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:03.588218 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 29 16:23:03.588666 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:03.588271 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 29 16:23:04.589160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:04.589121 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 29 16:23:04.589569 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:04.589182 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 29 16:23:05.588215 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:05.588178 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 29 16:23:05.588416 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:05.588240 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 29 16:23:06.591778 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:06.591748 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:23:06.592295 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:06.591999 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:23:06.592612 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:06.592594 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:23:11.698957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:11.698923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:12.037854 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.037825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:12.319241 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.319158 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:23:12.323186 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.323162 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.325827 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.325802 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8d3dc6-openshift-default-dockercfg-h5tw9\"" Jul 29 16:23:12.337228 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.337201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:23:12.352094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.352061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:12.408723 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdpq\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.408855 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.408855 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.408855 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.409009 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.409009 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.408917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.409094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.409014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.409094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.409062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.409094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.409078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdpq\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510542 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510833 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510833 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510833 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510716 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.510833 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.511066 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.510954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.511066 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.511037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.511415 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.511393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.512739 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.512716 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.512851 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.512832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.519651 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.519630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.520571 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.520529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdpq\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq\") pod \"e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.612855 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.612777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:12.636450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.636423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:12.784696 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.784667 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:23:12.786096 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:23:12.786073 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ece261_a9bd_4b7d_94a5_67d4ffe4ca45.slice/crio-645b30b50e938e28b02d511ba20ce5c06ebc2bb3f246cb1e7202450b6dabbefc WatchSource:0}: Error finding container 645b30b50e938e28b02d511ba20ce5c06ebc2bb3f246cb1e7202450b6dabbefc: Status 404 returned error can't find the container with id 645b30b50e938e28b02d511ba20ce5c06ebc2bb3f246cb1e7202450b6dabbefc Jul 29 16:23:12.788189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.788157 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:12.788272 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.788223 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:12.788272 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.788252 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:12.948998 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.948886 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerStarted","Data":"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1"} Jul 29 16:23:12.949151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.949008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerStarted","Data":"645b30b50e938e28b02d511ba20ce5c06ebc2bb3f246cb1e7202450b6dabbefc"} Jul 29 16:23:12.972051 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:12.972001 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podStartSLOduration=0.971971287 podStartE2EDuration="971.971287ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:23:12.969861029 +0000 UTC m=+2189.208418240" watchObservedRunningTime="2026-07-29 16:23:12.971971287 +0000 UTC m=+2189.210528497" Jul 29 16:23:13.637189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:13.637158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:13.642114 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:13.642089 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:13.953968 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:13.953944 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:13.954934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:13.954918 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:14.908801 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:14.908767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:14.998995 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:14.998945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:15.211572 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:15.211542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:15.403756 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:15.403714 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:15.689185 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:15.689100 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:15.786716 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:15.786684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:28.055346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.055270 2574 generic.go:356] "Generic (PLEG): container finished" podID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerID="7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1" exitCode=0 Jul 29 16:23:28.055740 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.055342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerDied","Data":"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1"} Jul 29 16:23:28.055803 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.055789 2574 scope.go:117] "RemoveContainer" containerID="7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1" Jul 29 16:23:28.056820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.056793 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:28.056879 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.056860 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:28.056914 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.056889 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:23:28.954224 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:28.954172 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:29.066332 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:29.066293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerStarted","Data":"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48"} Jul 29 16:23:29.637652 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:29.637607 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:29.638946 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:29.638917 2574 patch_prober.go:28] interesting pod/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 16:23:29.639080 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:29.639020 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 16:23:30.637691 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:30.637654 2574 patch_prober.go:28] interesting pod/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 16:23:30.638146 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:30.637725 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 16:23:31.637845 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:31.637808 2574 patch_prober.go:28] interesting pod/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 16:23:31.638283 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:31.637880 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 16:23:32.637116 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:32.637076 2574 patch_prober.go:28] interesting pod/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 29 16:23:32.637308 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:32.637149 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 29 16:23:33.641043 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:33.641008 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:33.641456 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:33.641266 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:33.641887 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:33.641870 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:23:45.011102 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:45.011065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:23:51.800839 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:23:51.800804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:24:02.718988 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:24:02.718940 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:24:16.190519 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:24:16.190441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:24:44.559415 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:24:44.559373 2574 scope.go:117] "RemoveContainer" containerID="4c91c77bac7ad33fb0105502eecc1d4fedf1936244b60cc4f5dfb7c9cfd6639f" Jul 29 16:24:59.549030 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:24:59.548967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:00.753984 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.753931 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f"] Jul 29 16:25:00.757419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.757402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:00.760628 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.760586 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 16:25:00.760851 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.760834 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:25:00.767804 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.767771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f"] Jul 29 16:25:00.861956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.861915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f\" (UID: \"5fc014c6-b80c-426c-afb5-fd16951df309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:00.963060 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.963027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f\" (UID: \"5fc014c6-b80c-426c-afb5-fd16951df309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:00.963665 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:00.963643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f\" (UID: \"5fc014c6-b80c-426c-afb5-fd16951df309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:01.069658 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:01.069568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:01.207616 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:01.207585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f"] Jul 29 16:25:01.211352 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:25:01.211326 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc014c6_b80c_426c_afb5_fd16951df309.slice/crio-6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773 WatchSource:0}: Error finding container 6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773: Status 404 returned error can't find the container with id 6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773 Jul 29 16:25:01.545359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:01.545322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" event={"ID":"5fc014c6-b80c-426c-afb5-fd16951df309","Type":"ContainerStarted","Data":"8f1857df0194bfd75bf57a8a6ac2228c8ae1588eecb0ccc9de524198a189975a"} Jul 29 16:25:01.545554 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:01.545382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" event={"ID":"5fc014c6-b80c-426c-afb5-fd16951df309","Type":"ContainerStarted","Data":"6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773"} Jul 29 16:25:01.578189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:01.578137 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" podStartSLOduration=1.578122514 podStartE2EDuration="1.578122514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:25:01.576971128 +0000 UTC m=+2297.815528338" watchObservedRunningTime="2026-07-29 16:25:01.578122514 +0000 UTC m=+2297.816679723" Jul 29 16:25:02.552369 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:02.552334 2574 generic.go:356] "Generic (PLEG): container finished" podID="5fc014c6-b80c-426c-afb5-fd16951df309" containerID="8f1857df0194bfd75bf57a8a6ac2228c8ae1588eecb0ccc9de524198a189975a" exitCode=0 Jul 29 16:25:02.552827 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:02.552421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" event={"ID":"5fc014c6-b80c-426c-afb5-fd16951df309","Type":"ContainerDied","Data":"8f1857df0194bfd75bf57a8a6ac2228c8ae1588eecb0ccc9de524198a189975a"} Jul 29 16:25:02.609110 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:02.609075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:03.703628 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:03.703604 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:03.790770 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:03.790738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca\") pod \"5fc014c6-b80c-426c-afb5-fd16951df309\" (UID: \"5fc014c6-b80c-426c-afb5-fd16951df309\") " Jul 29 16:25:03.791178 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:03.791145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fc014c6-b80c-426c-afb5-fd16951df309" (UID: "5fc014c6-b80c-426c-afb5-fd16951df309"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:03.892563 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:03.892474 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc014c6-b80c-426c-afb5-fd16951df309-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:04.570282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:04.570248 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" Jul 29 16:25:04.570455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:04.570270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f" event={"ID":"5fc014c6-b80c-426c-afb5-fd16951df309","Type":"ContainerDied","Data":"6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773"} Jul 29 16:25:04.570455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:04.570356 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a419e634af1b8743b732911e106a3c9311c82b252cd906e69d40187afb1e773" Jul 29 16:25:09.910357 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:09.910321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:11.257294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.257261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:11.280461 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.280428 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:25:11.280654 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.280629 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" podUID="df1502de-de66-4d4e-9341-ad12b643468e" containerName="payload-processing" containerID="cri-o://cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e" gracePeriod=30 Jul 29 16:25:11.357310 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.357248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:11.453004 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.450076 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f"] Jul 29 16:25:11.453004 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.451604 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:11.462874 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.462768 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b340d6-78l9f"] Jul 29 16:25:11.515391 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.515303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:11.573298 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.573274 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:25:11.611603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.611569 2574 generic.go:356] "Generic (PLEG): container finished" podID="df1502de-de66-4d4e-9341-ad12b643468e" containerID="cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e" exitCode=0 Jul 29 16:25:11.611771 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.611607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" Jul 29 16:25:11.611771 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.611614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" event={"ID":"df1502de-de66-4d4e-9341-ad12b643468e","Type":"ContainerDied","Data":"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e"} Jul 29 16:25:11.611771 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.611697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5" event={"ID":"df1502de-de66-4d4e-9341-ad12b643468e","Type":"ContainerDied","Data":"eb164557829b1d6f797f4282f20bfe653963b5f85c18945219b5ff5482457656"} Jul 29 16:25:11.611771 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.611729 2574 scope.go:117] "RemoveContainer" containerID="cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e" Jul 29 16:25:11.620215 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.620193 2574 scope.go:117] "RemoveContainer" containerID="cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e" Jul 29 16:25:11.620465 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:25:11.620445 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e\": container with ID starting with cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e not found: ID does not exist" containerID="cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e" Jul 29 16:25:11.620529 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.620479 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e"} err="failed to get container status \"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e\": rpc error: code = NotFound desc = could not find container \"cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e\": container with ID starting with cf5a30572c3e2dc7d6c91ac3ae506877daa91c73a7b011309b8a0881ed2e5c8e not found: ID does not exist" Jul 29 16:25:11.674006 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.673973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gqt8\" (UniqueName: \"kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8\") pod \"df1502de-de66-4d4e-9341-ad12b643468e\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " Jul 29 16:25:11.674138 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.674101 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume\") pod \"df1502de-de66-4d4e-9341-ad12b643468e\" (UID: \"df1502de-de66-4d4e-9341-ad12b643468e\") " Jul 29 16:25:11.674431 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.674409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df1502de-de66-4d4e-9341-ad12b643468e" (UID: "df1502de-de66-4d4e-9341-ad12b643468e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:11.675911 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.675881 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8" (OuterVolumeSpecName: "kube-api-access-9gqt8") pod "df1502de-de66-4d4e-9341-ad12b643468e" (UID: "df1502de-de66-4d4e-9341-ad12b643468e"). InnerVolumeSpecName "kube-api-access-9gqt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:11.775536 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.775460 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqt8\" (UniqueName: \"kubernetes.io/projected/df1502de-de66-4d4e-9341-ad12b643468e-kube-api-access-9gqt8\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:11.775536 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.775489 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df1502de-de66-4d4e-9341-ad12b643468e-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:11.936575 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.936547 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:25:11.943412 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:11.943386 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b340d6-647ccbd6fb-jmdk5"] Jul 29 16:25:12.096669 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:12.096563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:12.334161 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:12.334132 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fc014c6-b80c-426c-afb5-fd16951df309" path="/var/lib/kubelet/pods/5fc014c6-b80c-426c-afb5-fd16951df309/volumes" Jul 29 16:25:12.334557 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:12.334544 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df1502de-de66-4d4e-9341-ad12b643468e" path="/var/lib/kubelet/pods/df1502de-de66-4d4e-9341-ad12b643468e/volumes" Jul 29 16:25:18.483717 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:18.483685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:18.616836 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:18.616807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:18.790491 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:18.790402 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:25:18.790683 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:18.790656 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" containerID="cri-o://671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d" gracePeriod=30 Jul 29 16:25:18.831465 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:18.831417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:19.014647 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:19.014607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:21.594957 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:21.594910 2574 patch_prober.go:28] interesting pod/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:25:21.595448 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:21.595005 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:25:24.014610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.014573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:24.040708 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.040689 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:25:24.148559 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.148476 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:24.198631 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198589 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7d8n\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.198825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.198825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.198825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.198825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198762 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.198825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198785 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.199158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198836 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.199158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.198922 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.199158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199034 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy\") pod \"1394028f-9993-4594-b79b-b84a4f895734\" (UID: \"1394028f-9993-4594-b79b-b84a4f895734\") " Jul 29 16:25:24.199158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:24.199158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199035 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:24.199418 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:24.199418 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199244 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:24.199527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199443 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1394028f-9993-4594-b79b-b84a4f895734-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.199527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199462 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.199527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199474 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.199527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.199487 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.200244 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.200218 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data" (OuterVolumeSpecName: "istio-data") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:24.201344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.201319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token" (OuterVolumeSpecName: "istio-token") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:24.201434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.201365 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:24.201434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.201364 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:25:24.201820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.201802 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n" (OuterVolumeSpecName: "kube-api-access-s7d8n") pod "1394028f-9993-4594-b79b-b84a4f895734" (UID: "1394028f-9993-4594-b79b-b84a4f895734"). InnerVolumeSpecName "kube-api-access-s7d8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:24.280660 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.280620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:24.300901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.300810 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.300901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.300841 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1394028f-9993-4594-b79b-b84a4f895734-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.300901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.300854 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.300901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.300870 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7d8n\" (UniqueName: \"kubernetes.io/projected/1394028f-9993-4594-b79b-b84a4f895734-kube-api-access-s7d8n\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.300901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.300883 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1394028f-9993-4594-b79b-b84a4f895734-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:24.448324 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.448231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:24.680678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.680644 2574 generic.go:356] "Generic (PLEG): container finished" podID="1394028f-9993-4594-b79b-b84a4f895734" containerID="671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d" exitCode=0 Jul 29 16:25:24.680853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.680704 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" Jul 29 16:25:24.680853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.680732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerDied","Data":"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d"} Jul 29 16:25:24.680853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.680793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9" event={"ID":"1394028f-9993-4594-b79b-b84a4f895734","Type":"ContainerDied","Data":"74d1c2fc982ed77da5830ab6c86da275f899ab80dae77f534419c073e89f730a"} Jul 29 16:25:24.680853 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.680818 2574 scope.go:117] "RemoveContainer" containerID="671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d" Jul 29 16:25:24.689901 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.689877 2574 scope.go:117] "RemoveContainer" containerID="d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80" Jul 29 16:25:24.699402 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.699375 2574 scope.go:117] "RemoveContainer" containerID="671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d" Jul 29 16:25:24.699710 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:25:24.699690 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d\": container with ID starting with 671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d not found: ID does not exist" containerID="671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d" Jul 29 16:25:24.699789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.699718 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d"} err="failed to get container status \"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d\": rpc error: code = NotFound desc = could not find container \"671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d\": container with ID starting with 671e85751d4841764cfaa47e9c5ac2f83f9009a918ac29c08e3b036893ba4c9d not found: ID does not exist" Jul 29 16:25:24.699789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.699737 2574 scope.go:117] "RemoveContainer" containerID="d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80" Jul 29 16:25:24.700035 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:25:24.700012 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80\": container with ID starting with d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80 not found: ID does not exist" containerID="d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80" Jul 29 16:25:24.700035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.700034 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80"} err="failed to get container status \"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80\": rpc error: code = NotFound desc = could not find container \"d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80\": container with ID starting with d2dd7231d57c844c2ae965f0eaca7702fd4df685278c5a456e876c78adc5ab80 not found: ID does not exist" Jul 29 16:25:24.719869 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.719842 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:25:24.769070 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.769038 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b340d6-openshift-default-5568d57d7b-zrvl9"] Jul 29 16:25:24.795969 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:24.795931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:26.333431 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:26.333401 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1394028f-9993-4594-b79b-b84a4f895734" path="/var/lib/kubelet/pods/1394028f-9993-4594-b79b-b84a4f895734/volumes" Jul 29 16:25:31.108445 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.108416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:31.204089 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.204054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:31.347676 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.347641 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:25:31.347897 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.347871 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" containerID="cri-o://645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48" gracePeriod=30 Jul 29 16:25:31.381009 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.380902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:31.626404 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.626372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:31.929829 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:31.929796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:32.205627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.205594 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206009 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc014c6-b80c-426c-afb5-fd16951df309" containerName="revoke-keys" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206021 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc014c6-b80c-426c-afb5-fd16951df309" containerName="revoke-keys" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206039 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206044 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206055 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df1502de-de66-4d4e-9341-ad12b643468e" containerName="payload-processing" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206061 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df1502de-de66-4d4e-9341-ad12b643468e" containerName="payload-processing" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206131 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc014c6-b80c-426c-afb5-fd16951df309" containerName="revoke-keys" Jul 29 16:25:32.206145 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206141 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:25:32.206570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206152 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:25:32.206570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.206164 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df1502de-de66-4d4e-9341-ad12b643468e" containerName="payload-processing" Jul 29 16:25:32.210666 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.210646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.213363 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.213342 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef8cba-openshift-default-dockercfg-qv6zl\"" Jul 29 16:25:32.226685 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.226660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:25:32.228835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.228802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:32.384823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.384784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58btw\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385024 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.384841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385024 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.384868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385024 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.384971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.385046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.385103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.385145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.385167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.385211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.385198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58btw\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.485956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.485952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.486388 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.486305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.486443 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.486388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.486503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.486441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.486503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.486451 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.486718 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.486698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.488291 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.488272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.488387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.488292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.490197 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.490174 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:32.495739 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.495716 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.497185 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.497166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58btw\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw\") pod \"e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.522314 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.522290 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:32.672387 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.672361 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:25:32.675234 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:25:32.675201 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbb7d33_7f7d_4119_87c5_7e3ec4f8e477.slice/crio-015c476147bb1a6c43269dbcdb574608129a6306a6023affb94450bd8f3360ff WatchSource:0}: Error finding container 015c476147bb1a6c43269dbcdb574608129a6306a6023affb94450bd8f3360ff: Status 404 returned error can't find the container with id 015c476147bb1a6c43269dbcdb574608129a6306a6023affb94450bd8f3360ff Jul 29 16:25:32.677861 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.677820 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:32.677942 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.677913 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:32.678000 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.677961 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:32.725360 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:32.725333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerStarted","Data":"015c476147bb1a6c43269dbcdb574608129a6306a6023affb94450bd8f3360ff"} Jul 29 16:25:33.643951 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:33.643913 2574 patch_prober.go:28] interesting pod/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:25:33.644350 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:33.644005 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:25:33.732170 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:33.732133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerStarted","Data":"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57"} Jul 29 16:25:33.756393 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:33.756338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podStartSLOduration=1.756324057 podStartE2EDuration="1.756324057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:25:33.754774064 +0000 UTC m=+2329.993331273" watchObservedRunningTime="2026-07-29 16:25:33.756324057 +0000 UTC m=+2329.994881266" Jul 29 16:25:34.522792 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:34.522756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:34.527412 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:34.527387 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:34.737832 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:34.737801 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:34.738842 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:34.738820 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:35.703330 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:35.703240 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:35.799456 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:35.799413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:35.994893 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:35.994837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:36.621909 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.621849 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:25:36.725053 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725021 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqdpq\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725103 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725138 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725192 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725273 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725300 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket\") pod \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\" (UID: \"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45\") " Jul 29 16:25:36.725562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:36.725710 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725589 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:25:36.725710 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725641 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:36.725819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725707 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.725819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.725732 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:36.726138 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.726083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data" (OuterVolumeSpecName: "istio-data") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:36.727490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.727464 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq" (OuterVolumeSpecName: "kube-api-access-pqdpq") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "kube-api-access-pqdpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:36.727580 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.727561 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token" (OuterVolumeSpecName: "istio-token") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:25:36.727890 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.727865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:25:36.728125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.728103 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" (UID: "38ece261-a9bd-4b7d-94a5-67d4ffe4ca45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:25:36.748965 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.748934 2574 generic.go:356] "Generic (PLEG): container finished" podID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerID="645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48" exitCode=0 Jul 29 16:25:36.749092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.749001 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" Jul 29 16:25:36.749092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.749012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerDied","Data":"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48"} Jul 29 16:25:36.749092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.749080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv" event={"ID":"38ece261-a9bd-4b7d-94a5-67d4ffe4ca45","Type":"ContainerDied","Data":"645b30b50e938e28b02d511ba20ce5c06ebc2bb3f246cb1e7202450b6dabbefc"} Jul 29 16:25:36.749264 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.749112 2574 scope.go:117] "RemoveContainer" containerID="645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48" Jul 29 16:25:36.758054 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.758035 2574 scope.go:117] "RemoveContainer" containerID="7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1" Jul 29 16:25:36.766792 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.766766 2574 scope.go:117] "RemoveContainer" containerID="645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48" Jul 29 16:25:36.767130 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:25:36.767103 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48\": container with ID starting with 645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48 not found: ID does not exist" containerID="645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48" Jul 29 16:25:36.767242 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.767136 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48"} err="failed to get container status \"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48\": rpc error: code = NotFound desc = could not find container \"645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48\": container with ID starting with 645dc08a7fbe300c7e06ac479d136ac4f0ac099fb815230f390a78c3d3994b48 not found: ID does not exist" Jul 29 16:25:36.767242 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.767154 2574 scope.go:117] "RemoveContainer" containerID="7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1" Jul 29 16:25:36.767438 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:25:36.767413 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1\": container with ID starting with 7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1 not found: ID does not exist" containerID="7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1" Jul 29 16:25:36.767493 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.767447 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1"} err="failed to get container status \"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1\": rpc error: code = NotFound desc = could not find container \"7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1\": container with ID starting with 7b0a83cc825b4d7237fd670072deca7603267b9013aba0488ee23584c910afb1 not found: ID does not exist" Jul 29 16:25:36.790165 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.790100 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:36.793285 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.792440 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:25:36.797914 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.797883 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8d3dc6-openshift-default-d6477d876-rz4qv"] Jul 29 16:25:36.827032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827009 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827029 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827040 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827048 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827056 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdpq\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-kube-api-access-pqdpq\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827065 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827073 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.827434 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.827080 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:25:36.893321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:36.893223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:37.009089 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:37.009056 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:25:38.333255 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:38.333226 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" path="/var/lib/kubelet/pods/38ece261-a9bd-4b7d-94a5-67d4ffe4ca45/volumes" Jul 29 16:25:50.779141 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:50.779103 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 16:25:50.779582 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:50.779170 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 16:25:51.844156 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.844122 2574 generic.go:356] "Generic (PLEG): container finished" podID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerID="d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57" exitCode=0 Jul 29 16:25:51.844586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.844197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerDied","Data":"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57"} Jul 29 16:25:51.844586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.844579 2574 scope.go:117] "RemoveContainer" containerID="d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57" Jul 29 16:25:51.845568 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.845541 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:51.845617 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.845609 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:51.845653 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:51.845640 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:25:52.851220 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:52.851186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerStarted","Data":"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d"} Jul 29 16:25:53.522521 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:53.522482 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:53.522747 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:53.522725 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 29 16:25:53.522800 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:53.522771 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 29 16:25:54.523384 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:54.523342 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 29 16:25:54.523774 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:54.523398 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 29 16:25:55.523005 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:55.522944 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 29 16:25:55.523186 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:55.523029 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 29 16:25:56.522804 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:56.522758 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 29 16:25:56.523211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:56.522845 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 29 16:25:57.526894 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:57.526856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:57.527295 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:57.527093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:25:57.527757 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:25:57.527739 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:26:00.661010 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:00.660962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:01.144506 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.144468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:01.431449 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431358 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:26:01.431830 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:01.431919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:01.431919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431890 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:01.431919 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431901 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:01.432102 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431926 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:26:01.432102 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.431935 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1394028f-9993-4594-b79b-b84a4f895734" containerName="istio-proxy" Jul 29 16:26:01.432102 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.432059 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:01.434568 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.434547 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.437379 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.437360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6d4da6-openshift-default-dockercfg-vg2cw\"" Jul 29 16:26:01.451830 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.451802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:26:01.473001 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.472939 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:01.588649 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588649 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf6m\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.588865 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.589073 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.589073 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.588905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf6m\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.690768 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.691181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.691181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.691181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.690947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.691181 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.691004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.691394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.691256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.692947 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.692914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.693411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.693391 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.707533 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.707504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf6m\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.707695 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.707587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token\") pod \"e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.746466 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.746428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:01.861316 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.861283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:01.892955 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.892927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:26:01.895847 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:26:01.895813 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69784a71_4902_49ee_98df_5f2957da888e.slice/crio-48d015f04110650ddfff4fcae48eaec07c73d78afee4ee9ce7f866e8e509be88 WatchSource:0}: Error finding container 48d015f04110650ddfff4fcae48eaec07c73d78afee4ee9ce7f866e8e509be88: Status 404 returned error can't find the container with id 48d015f04110650ddfff4fcae48eaec07c73d78afee4ee9ce7f866e8e509be88 Jul 29 16:26:01.898242 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.898211 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:01.898349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.898291 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:01.898349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:01.898336 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:02.904010 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:02.903960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerStarted","Data":"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a"} Jul 29 16:26:02.904399 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:02.904036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerStarted","Data":"48d015f04110650ddfff4fcae48eaec07c73d78afee4ee9ce7f866e8e509be88"} Jul 29 16:26:02.939908 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:02.939854 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podStartSLOduration=1.939835373 podStartE2EDuration="1.939835373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:26:02.937561704 +0000 UTC m=+2359.176118907" watchObservedRunningTime="2026-07-29 16:26:02.939835373 +0000 UTC m=+2359.178392633" Jul 29 16:26:03.746810 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:03.746776 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:03.751787 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:03.751763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:03.910456 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:03.910426 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:03.911587 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:03.911573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:04.910531 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:04.910503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:05.048854 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:05.048813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:05.233189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:05.233155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:06.150488 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.150436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:06.281856 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.281822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:06.422426 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.422346 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:26:06.422792 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.422778 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ece261-a9bd-4b7d-94a5-67d4ffe4ca45" containerName="istio-proxy" Jul 29 16:26:06.424913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.424897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.427953 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.427921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6d4da6\"" Jul 29 16:26:06.428072 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.428043 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6d4da6-dockercfg-psj6z\"" Jul 29 16:26:06.440758 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.440728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8r2\" (UniqueName: \"kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.440917 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.440806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.452737 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.452705 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:26:06.541806 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.541768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.542002 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.541865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8r2\" (UniqueName: \"kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.542564 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.542541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.566405 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.566381 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8r2\" (UniqueName: \"kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2\") pod \"payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.736060 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.736021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:06.929507 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:06.929475 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:26:06.931175 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:26:06.931146 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e500336_3840_494e_b3b9_44b941d50660.slice/crio-725455a7ebf23f2c0bf0dcb060ccd1af8bd3cdae926e1c8c14eb43400e552ce7 WatchSource:0}: Error finding container 725455a7ebf23f2c0bf0dcb060ccd1af8bd3cdae926e1c8c14eb43400e552ce7: Status 404 returned error can't find the container with id 725455a7ebf23f2c0bf0dcb060ccd1af8bd3cdae926e1c8c14eb43400e552ce7 Jul 29 16:26:07.937292 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:07.937230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" event={"ID":"5e500336-3840-494e-b3b9-44b941d50660","Type":"ContainerStarted","Data":"c5c248b91e19edd3fd58e9f217f0f5d24caf2bd3d1d2a9b32eaf733aeb19db16"} Jul 29 16:26:07.937790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:07.937313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" event={"ID":"5e500336-3840-494e-b3b9-44b941d50660","Type":"ContainerStarted","Data":"725455a7ebf23f2c0bf0dcb060ccd1af8bd3cdae926e1c8c14eb43400e552ce7"} Jul 29 16:26:07.937790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:07.937361 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:07.962652 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:07.962604 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" podStartSLOduration=1.962586426 podStartE2EDuration="1.962586426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:26:07.961636521 +0000 UTC m=+2364.200193732" watchObservedRunningTime="2026-07-29 16:26:07.962586426 +0000 UTC m=+2364.201143654" Jul 29 16:26:18.944525 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:18.944493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:26:19.951169 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:19.951117 2574 patch_prober.go:28] interesting pod/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 16:26:19.951657 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:19.951197 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 16:26:22.014334 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.014298 2574 generic.go:356] "Generic (PLEG): container finished" podID="69784a71-4902-49ee-98df-5f2957da888e" containerID="7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a" exitCode=0 Jul 29 16:26:22.014782 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.014371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerDied","Data":"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a"} Jul 29 16:26:22.014854 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.014837 2574 scope.go:117] "RemoveContainer" containerID="7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a" Jul 29 16:26:22.015943 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.015911 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:22.016056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.015999 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:22.016056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:22.016051 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:26:23.022817 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:23.022788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerStarted","Data":"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250"} Jul 29 16:26:23.747508 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:23.747464 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:23.747726 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:23.747703 2574 patch_prober.go:28] interesting pod/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 29 16:26:23.747773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:23.747754 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 29 16:26:24.746861 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:24.746828 2574 patch_prober.go:28] interesting pod/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 29 16:26:24.747268 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:24.746885 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 29 16:26:25.747109 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:25.747071 2574 patch_prober.go:28] interesting pod/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 29 16:26:25.747500 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:25.747129 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 29 16:26:26.747363 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:26.747326 2574 patch_prober.go:28] interesting pod/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 29 16:26:26.747828 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:26.747387 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 29 16:26:27.751056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:27.751019 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:27.751455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:27.751312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:27.751992 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:27.751956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:26:29.889953 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:29.889915 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:44.489264 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:44.489237 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:26:44.509179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:44.509156 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:26:44.600735 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:44.600708 2574 scope.go:117] "RemoveContainer" containerID="0962c69225acc689652db9e8f4519c13e3ab69e06033880eb857ecf48b22e696" Jul 29 16:26:48.675871 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:48.675836 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:48.766602 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:48.766567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:48.835655 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:48.835623 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:48.906268 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:48.906235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:49.260246 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:49.260210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:56.033323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.033286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:56.134035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.134002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:56.287688 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.287583 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:26:56.287870 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.287847 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" containerID="cri-o://fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d" gracePeriod=30 Jul 29 16:26:56.304349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.304316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:56.531826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:56.531787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:26:57.530309 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:57.530268 2574 patch_prober.go:28] interesting pod/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 16:26:57.530754 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:26:57.530348 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 16:27:01.548232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.548207 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:27:01.671299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671212 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671265 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671299 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58btw\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671525 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671566 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671613 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671593 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671665 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671628 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671721 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:01.671721 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671668 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:01.671721 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671668 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671872 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671741 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy\") pod \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\" (UID: \"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477\") " Jul 29 16:27:01.671929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.671904 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:01.672035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672017 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:27:01.672211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672190 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.672211 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672209 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.672406 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672219 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.672406 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672228 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.672406 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.672191 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:01.673700 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.673675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw" (OuterVolumeSpecName: "kube-api-access-58btw") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "kube-api-access-58btw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:01.673809 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.673758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:27:01.674118 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.674099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:01.674118 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.674105 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" (UID: "2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:01.773288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.773253 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.773288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.773283 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.773288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.773291 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.773288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.773300 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:01.773560 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:01.773309 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58btw\" (UniqueName: \"kubernetes.io/projected/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477-kube-api-access-58btw\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:02.224218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.224186 2574 generic.go:356] "Generic (PLEG): container finished" podID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerID="fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d" exitCode=0 Jul 29 16:27:02.224398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.224237 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" Jul 29 16:27:02.224398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.224245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerDied","Data":"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d"} Jul 29 16:27:02.224398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.224288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg" event={"ID":"2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477","Type":"ContainerDied","Data":"015c476147bb1a6c43269dbcdb574608129a6306a6023affb94450bd8f3360ff"} Jul 29 16:27:02.224398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.224310 2574 scope.go:117] "RemoveContainer" containerID="fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d" Jul 29 16:27:02.233795 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.233778 2574 scope.go:117] "RemoveContainer" containerID="d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57" Jul 29 16:27:02.242036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.242020 2574 scope.go:117] "RemoveContainer" containerID="fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d" Jul 29 16:27:02.242264 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:27:02.242246 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d\": container with ID starting with fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d not found: ID does not exist" containerID="fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d" Jul 29 16:27:02.242322 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.242274 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d"} err="failed to get container status \"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d\": rpc error: code = NotFound desc = could not find container \"fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d\": container with ID starting with fac6ad4bda3e0762eb35dedc0286a8c8f57e40d43c5c540bfbbb5fabd5a7e45d not found: ID does not exist" Jul 29 16:27:02.242322 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.242289 2574 scope.go:117] "RemoveContainer" containerID="d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57" Jul 29 16:27:02.242478 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:27:02.242462 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57\": container with ID starting with d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57 not found: ID does not exist" containerID="d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57" Jul 29 16:27:02.242518 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.242481 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57"} err="failed to get container status \"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57\": rpc error: code = NotFound desc = could not find container \"d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57\": container with ID starting with d834bfedcda5ffa4dc278b146d902b0f317d87286cc7c694e8607ccc1e2e7f57 not found: ID does not exist" Jul 29 16:27:02.248165 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.248144 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:27:02.252453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.252434 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef8cba-openshift-default-7c478ccbf8-9qkgg"] Jul 29 16:27:02.333076 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:02.333035 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" path="/var/lib/kubelet/pods/2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477/volumes" Jul 29 16:27:07.001300 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.001198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:07.029867 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.029804 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:27:07.030349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.030101 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" podUID="5e500336-3840-494e-b3b9-44b941d50660" containerName="payload-processing" containerID="cri-o://c5c248b91e19edd3fd58e9f217f0f5d24caf2bd3d1d2a9b32eaf733aeb19db16" gracePeriod=30 Jul 29 16:27:07.073483 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.073453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:07.173956 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.173927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:07.250677 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.250645 2574 generic.go:356] "Generic (PLEG): container finished" podID="5e500336-3840-494e-b3b9-44b941d50660" containerID="c5c248b91e19edd3fd58e9f217f0f5d24caf2bd3d1d2a9b32eaf733aeb19db16" exitCode=0 Jul 29 16:27:07.250848 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.250688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" event={"ID":"5e500336-3840-494e-b3b9-44b941d50660","Type":"ContainerDied","Data":"c5c248b91e19edd3fd58e9f217f0f5d24caf2bd3d1d2a9b32eaf733aeb19db16"} Jul 29 16:27:07.271240 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.271123 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:07.292162 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.292140 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:27:07.423197 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.423163 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume\") pod \"5e500336-3840-494e-b3b9-44b941d50660\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " Jul 29 16:27:07.423362 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.423230 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx8r2\" (UniqueName: \"kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2\") pod \"5e500336-3840-494e-b3b9-44b941d50660\" (UID: \"5e500336-3840-494e-b3b9-44b941d50660\") " Jul 29 16:27:07.423503 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.423481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e500336-3840-494e-b3b9-44b941d50660" (UID: "5e500336-3840-494e-b3b9-44b941d50660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:27:07.425271 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.425243 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2" (OuterVolumeSpecName: "kube-api-access-zx8r2") pod "5e500336-3840-494e-b3b9-44b941d50660" (UID: "5e500336-3840-494e-b3b9-44b941d50660"). InnerVolumeSpecName "kube-api-access-zx8r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:07.524463 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.524386 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e500336-3840-494e-b3b9-44b941d50660-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:07.524463 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.524409 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8r2\" (UniqueName: \"kubernetes.io/projected/5e500336-3840-494e-b3b9-44b941d50660-kube-api-access-zx8r2\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:07.559333 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:07.559300 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:08.257453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.257419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" event={"ID":"5e500336-3840-494e-b3b9-44b941d50660","Type":"ContainerDied","Data":"725455a7ebf23f2c0bf0dcb060ccd1af8bd3cdae926e1c8c14eb43400e552ce7"} Jul 29 16:27:08.257453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.257456 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz" Jul 29 16:27:08.257946 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.257482 2574 scope.go:117] "RemoveContainer" containerID="c5c248b91e19edd3fd58e9f217f0f5d24caf2bd3d1d2a9b32eaf733aeb19db16" Jul 29 16:27:08.279541 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.279516 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:27:08.283264 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.283240 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6d4da6-fffd56c5b-nhhpz"] Jul 29 16:27:08.333272 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:08.333243 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e500336-3840-494e-b3b9-44b941d50660" path="/var/lib/kubelet/pods/5e500336-3840-494e-b3b9-44b941d50660/volumes" Jul 29 16:27:13.583673 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:13.583639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:13.646524 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:13.646495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:13.839159 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:13.839069 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:27:13.839346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:13.839314 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" containerID="cri-o://18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250" gracePeriod=30 Jul 29 16:27:14.282323 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.282290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:14.342994 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.342949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:14.411838 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.411805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:14.815356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815322 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:27:14.815790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815702 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:14.815790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815714 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:14.815790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815722 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e500336-3840-494e-b3b9-44b941d50660" containerName="payload-processing" Jul 29 16:27:14.815790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815728 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e500336-3840-494e-b3b9-44b941d50660" containerName="payload-processing" Jul 29 16:27:14.815926 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815817 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e500336-3840-494e-b3b9-44b941d50660" containerName="payload-processing" Jul 29 16:27:14.815926 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815829 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:14.815926 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.815835 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:14.819791 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.819772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.824366 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.824348 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-81ab8b-openshift-default-dockercfg-m4r7t\"" Jul 29 16:27:14.839610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.839577 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:27:14.890739 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.890934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.890934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6g45\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.890934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.890934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.891222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.890938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.891222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.891005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.891222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.891043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.891222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.891077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.925026 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.924960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:14.992543 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992543 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6g45\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.992813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993094 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.992871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993201 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.993117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993283 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.993258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993439 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.993419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993501 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.993441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.993557 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.993506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.995279 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.995254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:14.995748 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:14.995732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:15.003030 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.003003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:15.004295 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.004273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6g45\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45\") pod \"e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:15.078013 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.077909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:15.130282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.130240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:15.303296 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.303271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:27:15.305303 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:27:15.305276 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aef03e_3024_4281_a649_f425d6aebf15.slice/crio-a4fbdd7d49b8b1241798af8e6fb54e9e8b2ff257f09a19baf2b37dea75614c56 WatchSource:0}: Error finding container a4fbdd7d49b8b1241798af8e6fb54e9e8b2ff257f09a19baf2b37dea75614c56: Status 404 returned error can't find the container with id a4fbdd7d49b8b1241798af8e6fb54e9e8b2ff257f09a19baf2b37dea75614c56 Jul 29 16:27:15.307401 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.307366 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:15.307486 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.307430 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:15.307486 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:15.307458 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:16.303264 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:16.303215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerStarted","Data":"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e"} Jul 29 16:27:16.303734 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:16.303326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerStarted","Data":"a4fbdd7d49b8b1241798af8e6fb54e9e8b2ff257f09a19baf2b37dea75614c56"} Jul 29 16:27:16.328859 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:16.328797 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podStartSLOduration=2.3287810110000002 podStartE2EDuration="2.328781011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:27:16.326669177 +0000 UTC m=+2432.565226387" watchObservedRunningTime="2026-07-29 16:27:16.328781011 +0000 UTC m=+2432.567338293" Jul 29 16:27:17.131082 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:17.131044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:17.136074 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:17.136051 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:17.308385 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:17.308349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:17.309231 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:17.309214 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:18.292858 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:18.292807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:18.376933 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:18.376898 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:18.463070 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:18.463039 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:18.874093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:18.874062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:19.056346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.056316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:19.114930 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.114871 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:27:19.151291 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.151256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:19.236944 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.236907 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.236957 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237077 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237151 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237113 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237155 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237219 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237265 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkf6m\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237305 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:19.237623 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy\") pod \"69784a71-4902-49ee-98df-5f2957da888e\" (UID: \"69784a71-4902-49ee-98df-5f2957da888e\") " Jul 29 16:27:19.237623 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237433 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:19.237623 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237488 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:27:19.237801 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237717 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.237801 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237734 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.237801 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237747 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69784a71-4902-49ee-98df-5f2957da888e-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.237801 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.237760 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.238164 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.238140 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data" (OuterVolumeSpecName: "istio-data") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:19.239555 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.239524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m" (OuterVolumeSpecName: "kube-api-access-bkf6m") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "kube-api-access-bkf6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:19.239656 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.239613 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:27:19.239765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.239748 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:27:19.239818 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.239773 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token" (OuterVolumeSpecName: "istio-token") pod "69784a71-4902-49ee-98df-5f2957da888e" (UID: "69784a71-4902-49ee-98df-5f2957da888e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:27:19.322745 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.322710 2574 generic.go:356] "Generic (PLEG): container finished" podID="69784a71-4902-49ee-98df-5f2957da888e" containerID="18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250" exitCode=0 Jul 29 16:27:19.322963 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.322780 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" Jul 29 16:27:19.322963 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.322809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerDied","Data":"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250"} Jul 29 16:27:19.322963 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.322877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf" event={"ID":"69784a71-4902-49ee-98df-5f2957da888e","Type":"ContainerDied","Data":"48d015f04110650ddfff4fcae48eaec07c73d78afee4ee9ce7f866e8e509be88"} Jul 29 16:27:19.322963 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.322923 2574 scope.go:117] "RemoveContainer" containerID="18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250" Jul 29 16:27:19.332999 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.332958 2574 scope.go:117] "RemoveContainer" containerID="7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a" Jul 29 16:27:19.338537 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.338506 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.338634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.338539 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.338634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.338555 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkf6m\" (UniqueName: \"kubernetes.io/projected/69784a71-4902-49ee-98df-5f2957da888e-kube-api-access-bkf6m\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.338634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.338568 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69784a71-4902-49ee-98df-5f2957da888e-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.338634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.338580 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69784a71-4902-49ee-98df-5f2957da888e-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:27:19.342494 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.342476 2574 scope.go:117] "RemoveContainer" containerID="18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250" Jul 29 16:27:19.342747 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:27:19.342728 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250\": container with ID starting with 18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250 not found: ID does not exist" containerID="18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250" Jul 29 16:27:19.342811 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.342755 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250"} err="failed to get container status \"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250\": rpc error: code = NotFound desc = could not find container \"18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250\": container with ID starting with 18607768438d0e4aba21e54c08040ff49d6cf3a415f9d0a39b88bbe07e9b8250 not found: ID does not exist" Jul 29 16:27:19.342811 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.342772 2574 scope.go:117] "RemoveContainer" containerID="7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a" Jul 29 16:27:19.343051 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:27:19.343024 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a\": container with ID starting with 7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a not found: ID does not exist" containerID="7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a" Jul 29 16:27:19.343105 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.343060 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a"} err="failed to get container status \"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a\": rpc error: code = NotFound desc = could not find container \"7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a\": container with ID starting with 7f90248675668f107bb49b5fb6bff9c68bbfa4ebc1daac458567bea484cf708a not found: ID does not exist" Jul 29 16:27:19.351357 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.351337 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:27:19.356638 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:19.356613 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6d4da6-openshift-default-6c96c6f87b-hhmbf"] Jul 29 16:27:20.333428 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:20.333390 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69784a71-4902-49ee-98df-5f2957da888e" path="/var/lib/kubelet/pods/69784a71-4902-49ee-98df-5f2957da888e/volumes" Jul 29 16:27:31.340707 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.340662 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:31.394880 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.394840 2574 generic.go:356] "Generic (PLEG): container finished" podID="04aef03e-3024-4281-a649-f425d6aebf15" containerID="d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e" exitCode=0 Jul 29 16:27:31.395065 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.394905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerDied","Data":"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e"} Jul 29 16:27:31.395338 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.395325 2574 scope.go:117] "RemoveContainer" containerID="d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e" Jul 29 16:27:31.396421 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.396397 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:31.396481 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.396464 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:31.396519 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.396500 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:31.401411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:31.401384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:32.308577 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:32.308536 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:32.405570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:32.405527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerStarted","Data":"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8"} Jul 29 16:27:33.131356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:33.131310 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:33.132403 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:33.132378 2574 patch_prober.go:28] interesting pod/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 16:27:33.132530 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:33.132423 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 16:27:34.131121 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:34.131082 2574 patch_prober.go:28] interesting pod/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 16:27:34.131549 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:34.131148 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 16:27:35.131450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:35.131411 2574 patch_prober.go:28] interesting pod/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 16:27:35.131851 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:35.131481 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 16:27:36.131030 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:36.130962 2574 patch_prober.go:28] interesting pod/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 16:27:36.131207 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:36.131051 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 16:27:37.135049 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:37.135019 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:37.135433 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:37.135302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:37.135931 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:37.135915 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:27:43.317516 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.317475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:43.653342 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.653263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:43.967027 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.966964 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:27:43.967561 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:43.967610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbb7d33-7f7d-4119-87c5-7e3ec4f8e477" containerName="istio-proxy" Jul 29 16:27:43.967610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967581 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.967610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967590 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.967610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967601 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.967732 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967611 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.967732 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967699 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.967732 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.967709 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69784a71-4902-49ee-98df-5f2957da888e" containerName="istio-proxy" Jul 29 16:27:43.970623 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.970599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:43.973351 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.973329 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-eccd34-openshift-default-dockercfg-m4lj8\"" Jul 29 16:27:43.989431 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.989403 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:27:43.996488 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:43.996452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:44.095812 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.095766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096000 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.095852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cz7f\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096000 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.095890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096000 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.095914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.096031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.096083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.096127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.096159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.096261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.096188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197423 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cz7f\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197942 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197942 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.197942 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.197864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.198144 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.198041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.198627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.198339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.198627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.198492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.198627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.198577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.200334 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.200300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.200484 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.200468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.213821 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.213799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cz7f\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.217016 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.216992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token\") pod \"e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.284179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.284094 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-eccd34-openshift-default-dockercfg-m4lj8\"" Jul 29 16:27:44.292419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.292391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:44.369357 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.369314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:44.460586 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.460410 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:27:44.461458 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.461421 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:44.461573 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.461512 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:44.461689 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.461654 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:27:44.476756 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:44.476729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerStarted","Data":"df171a346c33ee307a358916a6bc8b3d6f71f42650044fcd2d7933784ff4883e"} Jul 29 16:27:45.483695 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:45.483659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerStarted","Data":"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b"} Jul 29 16:27:45.506001 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:45.505930 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podStartSLOduration=2.505913575 podStartE2EDuration="2.505913575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:27:45.503575783 +0000 UTC m=+2461.742132995" watchObservedRunningTime="2026-07-29 16:27:45.505913575 +0000 UTC m=+2461.744470785" Jul 29 16:27:46.293050 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:46.293008 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:46.298044 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:46.298020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:46.489085 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:46.489052 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:46.489929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:46.489910 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:27:47.571365 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:47.571330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:47.671421 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:47.671393 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:47.749562 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:47.749529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:47.966471 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:47.966437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:48.363923 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:48.363828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:48.464419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:48.464390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:27:53.142700 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:27:53.142671 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:28:02.531130 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:02.531085 2574 patch_prober.go:28] interesting pod/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 16:28:02.531665 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:02.531144 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 16:28:03.575197 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.575155 2574 generic.go:356] "Generic (PLEG): container finished" podID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerID="78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b" exitCode=0 Jul 29 16:28:03.575645 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.575232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerDied","Data":"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b"} Jul 29 16:28:03.575645 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.575627 2574 scope.go:117] "RemoveContainer" containerID="78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b" Jul 29 16:28:03.576675 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.576644 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:28:03.576777 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.576733 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:28:03.576834 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:03.576786 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:28:04.582287 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:04.582253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerStarted","Data":"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5"} Jul 29 16:28:05.293644 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:05.293607 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:28:05.293920 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:05.293658 2574 patch_prober.go:28] interesting pod/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 29 16:28:05.294114 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:05.294089 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 29 16:28:06.293501 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:06.293460 2574 patch_prober.go:28] interesting pod/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 29 16:28:06.294131 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:06.294097 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 29 16:28:07.293417 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:07.293380 2574 patch_prober.go:28] interesting pod/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 29 16:28:07.293716 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:07.293686 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 29 16:28:08.293939 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:08.293899 2574 patch_prober.go:28] interesting pod/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 29 16:28:08.294453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:08.293972 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 29 16:28:09.297116 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:09.297086 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:28:09.297570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:09.297362 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:28:09.298189 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:09.298169 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:28:12.494345 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:12.494312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:28:15.005101 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:15.005064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:28:23.296390 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:23.296345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:28:45.696140 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:28:45.696063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:30:00.136383 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.136346 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv"] Jul 29 16:30:00.139192 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.139172 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd"] Jul 29 16:30:00.139337 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.139316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" Jul 29 16:30:00.145014 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.142417 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n57pr\"" Jul 29 16:30:00.149311 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.149272 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch"] Jul 29 16:30:00.149449 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.149386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" Jul 29 16:30:00.153676 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.153653 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv"] Jul 29 16:30:00.154242 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.154199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch"] Jul 29 16:30:00.154375 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.154149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" Jul 29 16:30:00.158879 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.158861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" Jul 29 16:30:00.168232 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.168205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd"] Jul 29 16:30:00.168887 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.168790 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" Jul 29 16:30:00.202679 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.202655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" Jul 29 16:30:00.544571 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.544543 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv"] Jul 29 16:30:00.546541 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:30:00.546514 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2869b6f4_f1fe_4a8d_952a_9b58b5c64629.slice/crio-c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295 WatchSource:0}: Error finding container c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295: Status 404 returned error can't find the container with id c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295 Jul 29 16:30:00.569290 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.569262 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd"] Jul 29 16:30:00.574275 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:30:00.574251 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da9fa63_68a9_4705_bdaf_8b187d0fdbf6.slice/crio-050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89 WatchSource:0}: Error finding container 050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89: Status 404 returned error can't find the container with id 050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89 Jul 29 16:30:00.594020 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:00.593958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch"] Jul 29 16:30:00.603183 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:30:00.603150 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7997ee18_f2fb_42b5_8c0b_0be9651408f1.slice/crio-ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417 WatchSource:0}: Error finding container ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417: Status 404 returned error can't find the container with id ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417 Jul 29 16:30:01.171633 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.171519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" event={"ID":"2869b6f4-f1fe-4a8d-952a-9b58b5c64629","Type":"ContainerStarted","Data":"6608072eaa79500b9de60aea4d5651384425a0abc8f5377abcb48d5bf160d136"} Jul 29 16:30:01.171633 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.171593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" event={"ID":"2869b6f4-f1fe-4a8d-952a-9b58b5c64629","Type":"ContainerStarted","Data":"c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295"} Jul 29 16:30:01.173136 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.173109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" event={"ID":"7997ee18-f2fb-42b5-8c0b-0be9651408f1","Type":"ContainerStarted","Data":"3b97a0532eb649eed2021ebe7ceb6bd35f801b84c3d99991b6512e8934d5ca35"} Jul 29 16:30:01.173269 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.173153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" event={"ID":"7997ee18-f2fb-42b5-8c0b-0be9651408f1","Type":"ContainerStarted","Data":"ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417"} Jul 29 16:30:01.174539 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.174513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" event={"ID":"4da9fa63-68a9-4705-bdaf-8b187d0fdbf6","Type":"ContainerStarted","Data":"c40186b8788b005b380584ed3e46d2c8e3a93bf0f2bddf20f42e813e21edb69a"} Jul 29 16:30:01.174616 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.174557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" event={"ID":"4da9fa63-68a9-4705-bdaf-8b187d0fdbf6","Type":"ContainerStarted","Data":"050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89"} Jul 29 16:30:01.188356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.188311 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" podStartSLOduration=1.188296129 podStartE2EDuration="1.188296129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:30:01.185921532 +0000 UTC m=+2597.424478742" watchObservedRunningTime="2026-07-29 16:30:01.188296129 +0000 UTC m=+2597.426853338" Jul 29 16:30:01.199741 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.199687 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" podStartSLOduration=1.199632424 podStartE2EDuration="1.199632424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:30:01.199580672 +0000 UTC m=+2597.438137883" watchObservedRunningTime="2026-07-29 16:30:01.199632424 +0000 UTC m=+2597.438189638" Jul 29 16:30:01.213959 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:01.213890 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" podStartSLOduration=1.213868907 podStartE2EDuration="1.213868907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:30:01.21343562 +0000 UTC m=+2597.451992830" watchObservedRunningTime="2026-07-29 16:30:01.213868907 +0000 UTC m=+2597.452426117" Jul 29 16:30:02.181297 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.181260 2574 generic.go:356] "Generic (PLEG): container finished" podID="2869b6f4-f1fe-4a8d-952a-9b58b5c64629" containerID="6608072eaa79500b9de60aea4d5651384425a0abc8f5377abcb48d5bf160d136" exitCode=0 Jul 29 16:30:02.181739 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.181311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" event={"ID":"2869b6f4-f1fe-4a8d-952a-9b58b5c64629","Type":"ContainerDied","Data":"6608072eaa79500b9de60aea4d5651384425a0abc8f5377abcb48d5bf160d136"} Jul 29 16:30:02.182802 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.182779 2574 generic.go:356] "Generic (PLEG): container finished" podID="7997ee18-f2fb-42b5-8c0b-0be9651408f1" containerID="3b97a0532eb649eed2021ebe7ceb6bd35f801b84c3d99991b6512e8934d5ca35" exitCode=0 Jul 29 16:30:02.182934 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.182857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" event={"ID":"7997ee18-f2fb-42b5-8c0b-0be9651408f1","Type":"ContainerDied","Data":"3b97a0532eb649eed2021ebe7ceb6bd35f801b84c3d99991b6512e8934d5ca35"} Jul 29 16:30:02.184209 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.184189 2574 generic.go:356] "Generic (PLEG): container finished" podID="4da9fa63-68a9-4705-bdaf-8b187d0fdbf6" containerID="c40186b8788b005b380584ed3e46d2c8e3a93bf0f2bddf20f42e813e21edb69a" exitCode=0 Jul 29 16:30:02.184287 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:02.184219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" event={"ID":"4da9fa63-68a9-4705-bdaf-8b187d0fdbf6","Type":"ContainerDied","Data":"c40186b8788b005b380584ed3e46d2c8e3a93bf0f2bddf20f42e813e21edb69a"} Jul 29 16:30:03.343761 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:03.343735 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" Jul 29 16:30:03.379113 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:03.379087 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" Jul 29 16:30:03.382113 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:03.382091 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" Jul 29 16:30:04.199891 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.199852 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" Jul 29 16:30:04.200114 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.199852 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch" event={"ID":"7997ee18-f2fb-42b5-8c0b-0be9651408f1","Type":"ContainerDied","Data":"ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417"} Jul 29 16:30:04.200114 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.200060 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac0b5b4d4aa78e582c37ba0f831014ac0deecb2c716798c6e3e37a07eea51417" Jul 29 16:30:04.201149 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.201130 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" Jul 29 16:30:04.201273 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.201173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755710-blztd" event={"ID":"4da9fa63-68a9-4705-bdaf-8b187d0fdbf6","Type":"ContainerDied","Data":"050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89"} Jul 29 16:30:04.201273 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.201212 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="050b9f096c79c98b6e9fecb7cdb1555dd374760e3e1455a1a04aa990ba3f7a89" Jul 29 16:30:04.202443 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.202424 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" Jul 29 16:30:04.202541 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.202425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv" event={"ID":"2869b6f4-f1fe-4a8d-952a-9b58b5c64629","Type":"ContainerDied","Data":"c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295"} Jul 29 16:30:04.202607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:30:04.202557 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c63c26cf54cb60aa540d2c8b820963107910f5a5e460e2909a2b39043cb96295" Jul 29 16:31:44.525050 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:31:44.524946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:31:44.541264 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:31:44.541238 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:31:44.667529 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:31:44.667496 2574 scope.go:117] "RemoveContainer" containerID="8f1857df0194bfd75bf57a8a6ac2228c8ae1588eecb0ccc9de524198a189975a" Jul 29 16:32:33.831873 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:32:33.831827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:32:45.503364 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:32:45.503325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:19.772827 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:19.772738 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:20.674142 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:20.674101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:21.186025 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:21.185990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:21.287517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:21.287481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:33.958730 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:33.958693 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:33:54.547696 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:33:54.547659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:34:03.688237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:34:03.688199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:34:09.581943 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:34:09.581912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:34:09.681096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:34:09.681054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:35:05.756894 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:35:05.756852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:35:06.682023 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:35:06.681961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:35:16.704293 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:35:16.704234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:35:27.287288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:35:27.287236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:35:54.500847 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:35:54.500802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:36:44.555783 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:36:44.555754 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:36:44.572540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:36:44.572515 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:36:57.483726 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:36:57.483688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:36:58.469761 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:36:58.469713 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:02.761006 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.760948 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7654-139973292335152"] Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761358 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2869b6f4-f1fe-4a8d-952a-9b58b5c64629" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761371 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2869b6f4-f1fe-4a8d-952a-9b58b5c64629" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761379 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da9fa63-68a9-4705-bdaf-8b187d0fdbf6" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761385 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da9fa63-68a9-4705-bdaf-8b187d0fdbf6" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761393 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7997ee18-f2fb-42b5-8c0b-0be9651408f1" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761398 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7997ee18-f2fb-42b5-8c0b-0be9651408f1" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761455 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2869b6f4-f1fe-4a8d-952a-9b58b5c64629" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761463 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7997ee18-f2fb-42b5-8c0b-0be9651408f1" containerName="cleanup" Jul 29 16:37:02.761490 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.761470 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da9fa63-68a9-4705-bdaf-8b187d0fdbf6" containerName="cleanup" Jul 29 16:37:02.764555 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.764538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:02.768245 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.768086 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lqpzd\"" Jul 29 16:37:02.772158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.772126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292335152"] Jul 29 16:37:02.940321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:02.940276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54p97\" (UniqueName: \"kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97\") pod \"test-curl-7654-139973292335152\" (UID: \"81fd564b-6cfc-497d-9831-191f66f9d2c1\") " pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:03.041597 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.041504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54p97\" (UniqueName: \"kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97\") pod \"test-curl-7654-139973292335152\" (UID: \"81fd564b-6cfc-497d-9831-191f66f9d2c1\") " pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:03.051892 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.051862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54p97\" (UniqueName: \"kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97\") pod \"test-curl-7654-139973292335152\" (UID: \"81fd564b-6cfc-497d-9831-191f66f9d2c1\") " pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:03.078039 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.078002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:03.214890 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.214858 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292335152"] Jul 29 16:37:03.216492 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:37:03.216468 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81fd564b_6cfc_497d_9831_191f66f9d2c1.slice/crio-6c46653d5caa258c35235c57e0965a514c5848571045e092285e174f5908c231 WatchSource:0}: Error finding container 6c46653d5caa258c35235c57e0965a514c5848571045e092285e174f5908c231: Status 404 returned error can't find the container with id 6c46653d5caa258c35235c57e0965a514c5848571045e092285e174f5908c231 Jul 29 16:37:03.218157 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.218139 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 16:37:03.381001 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:03.380907 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292335152" event={"ID":"81fd564b-6cfc-497d-9831-191f66f9d2c1","Type":"ContainerStarted","Data":"6c46653d5caa258c35235c57e0965a514c5848571045e092285e174f5908c231"} Jul 29 16:37:04.387502 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:04.387469 2574 generic.go:356] "Generic (PLEG): container finished" podID="81fd564b-6cfc-497d-9831-191f66f9d2c1" containerID="d1838d78a7fcaa357fcffcd39ae1bf3e9603abeb1001a72bf27e665b918829e4" exitCode=0 Jul 29 16:37:04.387916 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:04.387564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292335152" event={"ID":"81fd564b-6cfc-497d-9831-191f66f9d2c1","Type":"ContainerDied","Data":"d1838d78a7fcaa357fcffcd39ae1bf3e9603abeb1001a72bf27e665b918829e4"} Jul 29 16:37:05.535453 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.535428 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:05.555405 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.555375 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7654-139973292335152_81fd564b-6cfc-497d-9831-191f66f9d2c1/test-curl-7654-139973292335152/0.log" Jul 29 16:37:05.561674 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.561634 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54p97\" (UniqueName: \"kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97\") pod \"81fd564b-6cfc-497d-9831-191f66f9d2c1\" (UID: \"81fd564b-6cfc-497d-9831-191f66f9d2c1\") " Jul 29 16:37:05.563832 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.563805 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97" (OuterVolumeSpecName: "kube-api-access-54p97") pod "81fd564b-6cfc-497d-9831-191f66f9d2c1" (UID: "81fd564b-6cfc-497d-9831-191f66f9d2c1"). InnerVolumeSpecName "kube-api-access-54p97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:37:05.575717 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.575687 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7654-139973292335152"] Jul 29 16:37:05.578718 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.578694 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7654-139973292335152"] Jul 29 16:37:05.662449 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:05.662360 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54p97\" (UniqueName: \"kubernetes.io/projected/81fd564b-6cfc-497d-9831-191f66f9d2c1-kube-api-access-54p97\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:37:06.334093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:06.334060 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fd564b-6cfc-497d-9831-191f66f9d2c1" path="/var/lib/kubelet/pods/81fd564b-6cfc-497d-9831-191f66f9d2c1/volumes" Jul 29 16:37:06.398852 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:06.398823 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292335152" Jul 29 16:37:06.399060 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:06.398876 2574 scope.go:117] "RemoveContainer" containerID="d1838d78a7fcaa357fcffcd39ae1bf3e9603abeb1001a72bf27e665b918829e4" Jul 29 16:37:11.914138 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.914102 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7654-139973292301312"] Jul 29 16:37:11.914517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.914501 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fd564b-6cfc-497d-9831-191f66f9d2c1" containerName="test-curl-7654-139973292335152" Jul 29 16:37:11.914517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.914513 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fd564b-6cfc-497d-9831-191f66f9d2c1" containerName="test-curl-7654-139973292335152" Jul 29 16:37:11.914602 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.914576 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fd564b-6cfc-497d-9831-191f66f9d2c1" containerName="test-curl-7654-139973292335152" Jul 29 16:37:11.918952 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.918935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:11.922059 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.922035 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lqpzd\"" Jul 29 16:37:11.924174 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:11.924147 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292301312"] Jul 29 16:37:12.023005 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.022949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4tt\" (UniqueName: \"kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt\") pod \"test-curl-7654-139973292301312\" (UID: \"5ae459ed-31ee-4eae-a9a9-20def778b642\") " pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:12.124314 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.124273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4tt\" (UniqueName: \"kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt\") pod \"test-curl-7654-139973292301312\" (UID: \"5ae459ed-31ee-4eae-a9a9-20def778b642\") " pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:12.132875 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.132844 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4tt\" (UniqueName: \"kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt\") pod \"test-curl-7654-139973292301312\" (UID: \"5ae459ed-31ee-4eae-a9a9-20def778b642\") " pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:12.229019 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.228960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:12.364564 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.364535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7654-139973292301312"] Jul 29 16:37:12.366836 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:37:12.366805 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae459ed_31ee_4eae_a9a9_20def778b642.slice/crio-531a83e1079183ea86cfe178c15b1db88b1f5c1b55709f133290919f6f94e29f WatchSource:0}: Error finding container 531a83e1079183ea86cfe178c15b1db88b1f5c1b55709f133290919f6f94e29f: Status 404 returned error can't find the container with id 531a83e1079183ea86cfe178c15b1db88b1f5c1b55709f133290919f6f94e29f Jul 29 16:37:12.441224 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:12.441191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292301312" event={"ID":"5ae459ed-31ee-4eae-a9a9-20def778b642","Type":"ContainerStarted","Data":"531a83e1079183ea86cfe178c15b1db88b1f5c1b55709f133290919f6f94e29f"} Jul 29 16:37:13.447862 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:13.447770 2574 generic.go:356] "Generic (PLEG): container finished" podID="5ae459ed-31ee-4eae-a9a9-20def778b642" containerID="cf31a6fd98ebe7672ce186af1c1e7e0697eb055100bb31a870cd265d42bc3c58" exitCode=0 Jul 29 16:37:13.447862 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:13.447825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7654-139973292301312" event={"ID":"5ae459ed-31ee-4eae-a9a9-20def778b642","Type":"ContainerDied","Data":"cf31a6fd98ebe7672ce186af1c1e7e0697eb055100bb31a870cd265d42bc3c58"} Jul 29 16:37:14.602676 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.602647 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:14.624545 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.624488 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7654-139973292301312_5ae459ed-31ee-4eae-a9a9-20def778b642/test-curl-7654-139973292301312/0.log" Jul 29 16:37:14.651699 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.651670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm4tt\" (UniqueName: \"kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt\") pod \"5ae459ed-31ee-4eae-a9a9-20def778b642\" (UID: \"5ae459ed-31ee-4eae-a9a9-20def778b642\") " Jul 29 16:37:14.653729 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.653702 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt" (OuterVolumeSpecName: "kube-api-access-xm4tt") pod "5ae459ed-31ee-4eae-a9a9-20def778b642" (UID: "5ae459ed-31ee-4eae-a9a9-20def778b642"). InnerVolumeSpecName "kube-api-access-xm4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:37:14.653967 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.653943 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7654-139973292301312"] Jul 29 16:37:14.659359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.659336 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7654-139973292301312"] Jul 29 16:37:14.753258 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:14.753227 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4tt\" (UniqueName: \"kubernetes.io/projected/5ae459ed-31ee-4eae-a9a9-20def778b642-kube-api-access-xm4tt\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:37:15.460270 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:15.460241 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531a83e1079183ea86cfe178c15b1db88b1f5c1b55709f133290919f6f94e29f" Jul 29 16:37:15.460465 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:15.460272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:15.471094 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:37:15.471066 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7654-139973292301312\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="5ae459ed-31ee-4eae-a9a9-20def778b642" pod="opendatahub/test-curl-7654-139973292301312" Jul 29 16:37:16.333040 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:16.333011 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae459ed-31ee-4eae-a9a9-20def778b642" path="/var/lib/kubelet/pods/5ae459ed-31ee-4eae-a9a9-20def778b642/volumes" Jul 29 16:37:18.222252 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.222217 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:18.596121 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.596041 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:37:18.596478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.596464 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae459ed-31ee-4eae-a9a9-20def778b642" containerName="test-curl-7654-139973292301312" Jul 29 16:37:18.596532 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.596482 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae459ed-31ee-4eae-a9a9-20def778b642" containerName="test-curl-7654-139973292301312" Jul 29 16:37:18.596577 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.596569 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae459ed-31ee-4eae-a9a9-20def778b642" containerName="test-curl-7654-139973292301312" Jul 29 16:37:18.601648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.601628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.605405 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.605386 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-777250-openshift-default-dockercfg-9mgwl\"" Jul 29 16:37:18.617113 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.617089 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:37:18.642716 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.642690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:18.687325 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687496 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687496 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687496 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687496 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.687765 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.687755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8x5n\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.759854 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.759812 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:18.789236 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8x5n\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789603 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789796 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.789914 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.790006 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.789967 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.790069 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.790004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.790257 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.790240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.792112 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.792088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.793481 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.793458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.800332 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.800312 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8x5n\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.802071 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.802050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token\") pod \"e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:18.912553 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:18.912475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:19.070952 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.070909 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:19.071100 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.071029 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:19.071100 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.071066 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:37:19.071206 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.071189 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:19.483285 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.483251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerStarted","Data":"7ab2f29cfb1e667bd7f262d4e40a218bb108e0876d33b85b91669271f525cddc"} Jul 29 16:37:19.483760 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.483313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerStarted","Data":"febda0df61d01789a7164263b40a74ffdbc56cf0dcc447b42f02c9a9f9680e8d"} Jul 29 16:37:19.516257 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.516194 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podStartSLOduration=1.51617325 podStartE2EDuration="1.51617325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:37:19.514026722 +0000 UTC m=+3035.752583932" watchObservedRunningTime="2026-07-29 16:37:19.51617325 +0000 UTC m=+3035.754730461" Jul 29 16:37:19.912840 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:19.912739 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:20.918517 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:20.918484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:21.494268 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:21.494237 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:21.495438 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:21.495416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:21.551775 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:21.551739 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:21.634191 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:21.634148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:21.816995 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:21.816868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:22.213492 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:22.213435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:22.612881 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:22.612773 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:22.718164 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:22.718122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:35.571930 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.571858 2574 generic.go:356] "Generic (PLEG): container finished" podID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerID="7ab2f29cfb1e667bd7f262d4e40a218bb108e0876d33b85b91669271f525cddc" exitCode=0 Jul 29 16:37:35.571930 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.571922 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerDied","Data":"7ab2f29cfb1e667bd7f262d4e40a218bb108e0876d33b85b91669271f525cddc"} Jul 29 16:37:35.572390 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.572373 2574 scope.go:117] "RemoveContainer" containerID="7ab2f29cfb1e667bd7f262d4e40a218bb108e0876d33b85b91669271f525cddc" Jul 29 16:37:35.573465 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.573436 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:35.573534 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.573521 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:35.573574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:35.573556 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:36.494637 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:36.494597 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:36.579288 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:36.579243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerStarted","Data":"c4b06038805fcace770120989e585f9a7e38e5fed3925183f8fa71671f6f9775"} Jul 29 16:37:36.913752 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:36.913610 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:36.915091 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:36.915054 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 29 16:37:36.915249 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:36.915119 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 29 16:37:37.913839 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:37.913800 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 29 16:37:37.914451 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:37.914421 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 29 16:37:38.913447 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:38.913408 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 29 16:37:38.914176 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:38.913743 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 29 16:37:39.914266 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:39.913791 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 29 16:37:39.914266 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:39.913858 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 29 16:37:40.917392 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:40.917362 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:40.917831 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:40.917759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:40.918446 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:40.918429 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:37:46.523254 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:46.523204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:47.543569 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.543534 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:37:47.547735 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.547708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.550748 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.550721 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3c2006-openshift-default-dockercfg-fx8xz\"" Jul 29 16:37:47.569474 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.569447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:37:47.589179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589138 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589441 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589441 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589441 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589441 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2k8v\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589441 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589689 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.589689 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.589510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690423 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690629 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690629 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690629 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690629 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.690819 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2k8v\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.691001 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.690895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.691627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.691069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.691627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.691299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.691627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.691393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.691627 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.691579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.693709 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.693678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.693837 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.693714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.713234 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.713204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2k8v\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.720646 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.720603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token\") pod \"e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:47.861073 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:47.860932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:48.010572 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:37:48.010542 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcf70e7_6e2c_43c6_9a9b_8ce1aa28ecb9.slice/crio-f5f669ce473b0cd5da0af425888d3bcd1faa4b7495a8bfdf0aefd6b2c8dd36d2 WatchSource:0}: Error finding container f5f669ce473b0cd5da0af425888d3bcd1faa4b7495a8bfdf0aefd6b2c8dd36d2: Status 404 returned error can't find the container with id f5f669ce473b0cd5da0af425888d3bcd1faa4b7495a8bfdf0aefd6b2c8dd36d2 Jul 29 16:37:48.010572 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.010551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:37:48.012805 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.012773 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:48.012897 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.012835 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:48.012897 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.012866 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:37:48.274773 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.274730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:48.671961 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.671869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerStarted","Data":"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a"} Jul 29 16:37:48.671961 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.671924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerStarted","Data":"f5f669ce473b0cd5da0af425888d3bcd1faa4b7495a8bfdf0aefd6b2c8dd36d2"} Jul 29 16:37:48.681972 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.681940 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:48.696629 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.696580 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podStartSLOduration=1.6965620449999999 podStartE2EDuration="1.696562045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:37:48.694317878 +0000 UTC m=+3064.932875088" watchObservedRunningTime="2026-07-29 16:37:48.696562045 +0000 UTC m=+3064.935119315" Jul 29 16:37:48.861117 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:48.861084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:49.867171 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:49.867141 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:50.646637 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:50.646601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:50.683282 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:50.683251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:50.684158 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:50.684137 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:37:50.847217 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:50.847187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:50.951104 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:50.951009 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:51.048652 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:51.048612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:51.474205 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:51.474168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:37:51.577398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:37:51.577364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:38:06.725190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:06.725145 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 16:38:06.725669 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:06.725237 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 16:38:08.783415 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.783382 2574 generic.go:356] "Generic (PLEG): container finished" podID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerID="95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a" exitCode=0 Jul 29 16:38:08.783825 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.783425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerDied","Data":"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a"} Jul 29 16:38:08.783873 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.783824 2574 scope.go:117] "RemoveContainer" containerID="95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a" Jul 29 16:38:08.785058 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.785029 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:38:08.785132 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.785118 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:38:08.785173 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:08.785153 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:38:09.790748 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:09.790710 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerStarted","Data":"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2"} Jul 29 16:38:09.861353 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:09.861320 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:38:09.861544 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:09.861524 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 29 16:38:09.861619 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:09.861593 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 29 16:38:10.861674 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:10.861642 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 29 16:38:10.862080 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:10.861693 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 29 16:38:11.862237 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:11.862203 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 29 16:38:11.862634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:11.862268 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 29 16:38:12.861738 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:12.861695 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 29 16:38:12.861949 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:12.861758 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 29 16:38:13.865056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:13.865021 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:38:13.865547 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:13.865423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:38:13.866028 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:13.866009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:38:20.814035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:20.813994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:38:33.571736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:38:33.571695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:15.173241 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.173132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:15.352341 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.352309 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:15.650743 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.650706 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:39:15.653947 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.653922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.669330 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.669303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1bad22-openshift-default-dockercfg-q7ksv\"" Jul 29 16:39:15.671993 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.671956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:39:15.676580 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.676558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:15.757799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.757760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758031 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.757834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758031 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.757920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758031 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.757963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758031 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.758013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.758040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zcg\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.758076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.758129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.758261 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.758177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859648 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859962 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.859962 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.859908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zcg\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860120 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860230 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860230 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860230 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860375 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.860497 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.860454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.862321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.862298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.862621 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.862604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.868329 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.868300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.868467 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.868449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zcg\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg\") pod \"e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.964456 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.964426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:15.975997 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:15.975927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:16.129068 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:16.129007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:39:16.132686 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:39:16.132642 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6779c7c9_a6fe_456e_bfed_157d35c57c41.slice/crio-23fedbfc3e8f401f74f520f5232fced6d6815615cc38c2934aedc1768698a3de WatchSource:0}: Error finding container 23fedbfc3e8f401f74f520f5232fced6d6815615cc38c2934aedc1768698a3de: Status 404 returned error can't find the container with id 23fedbfc3e8f401f74f520f5232fced6d6815615cc38c2934aedc1768698a3de Jul 29 16:39:16.135345 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:16.135314 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:16.135455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:16.135400 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:16.135455 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:16.135445 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:16.159797 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:16.159768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerStarted","Data":"23fedbfc3e8f401f74f520f5232fced6d6815615cc38c2934aedc1768698a3de"} Jul 29 16:39:17.167333 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:17.167300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerStarted","Data":"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81"} Jul 29 16:39:17.196430 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:17.196364 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podStartSLOduration=2.196343363 podStartE2EDuration="2.196343363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:39:17.191271758 +0000 UTC m=+3153.429828968" watchObservedRunningTime="2026-07-29 16:39:17.196343363 +0000 UTC m=+3153.434900575" Jul 29 16:39:17.965344 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:17.965308 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:17.969755 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:17.969735 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:18.172275 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:18.172245 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:18.173251 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:18.173233 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:19.121811 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.121767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:19.210757 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.210725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:19.295403 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.295368 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:19.400466 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.400281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:19.872294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.872264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:19.972268 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:19.972235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:20.146787 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.146704 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:20.150240 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.150217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.153494 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.153472 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1bad22-dockercfg-pmm4h\"" Jul 29 16:39:20.153615 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.153476 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1bad22\"" Jul 29 16:39:20.158737 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.158701 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:20.207770 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.207724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.207770 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.207772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6lh\" (UniqueName: \"kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.309167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.309133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.309167 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.309169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6lh\" (UniqueName: \"kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.310024 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.309969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.318758 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.318717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6lh\" (UniqueName: \"kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh\") pod \"payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.463178 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.463143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:20.614398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:20.614370 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:20.616553 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:39:20.616527 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217f7f7f_556a_405f_9d78_235eaf3d7150.slice/crio-8017ec8d3d64187fc6d7dc0270bda07f3407c0f9f6bf9aa650b4cb8e6dda9250 WatchSource:0}: Error finding container 8017ec8d3d64187fc6d7dc0270bda07f3407c0f9f6bf9aa650b4cb8e6dda9250: Status 404 returned error can't find the container with id 8017ec8d3d64187fc6d7dc0270bda07f3407c0f9f6bf9aa650b4cb8e6dda9250 Jul 29 16:39:21.190170 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:21.190132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" event={"ID":"217f7f7f-556a-405f-9d78-235eaf3d7150","Type":"ContainerStarted","Data":"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4"} Jul 29 16:39:21.190339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:21.190226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" event={"ID":"217f7f7f-556a-405f-9d78-235eaf3d7150","Type":"ContainerStarted","Data":"8017ec8d3d64187fc6d7dc0270bda07f3407c0f9f6bf9aa650b4cb8e6dda9250"} Jul 29 16:39:21.190339 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:21.190254 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:21.207931 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:21.207881 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" podStartSLOduration=1.207866403 podStartE2EDuration="1.207866403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 16:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:39:21.206271429 +0000 UTC m=+3157.444828640" watchObservedRunningTime="2026-07-29 16:39:21.207866403 +0000 UTC m=+3157.446423629" Jul 29 16:39:32.199139 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.199092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:32.258403 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.258372 2574 generic.go:356] "Generic (PLEG): container finished" podID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerID="92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81" exitCode=0 Jul 29 16:39:32.258599 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.258447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerDied","Data":"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81"} Jul 29 16:39:32.258944 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.258926 2574 scope.go:117] "RemoveContainer" containerID="92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81" Jul 29 16:39:32.260346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.260317 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:32.260437 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.260384 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:32.260437 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:32.260431 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 16:39:33.172790 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:33.172751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:33.265948 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:33.265910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerStarted","Data":"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae"} Jul 29 16:39:33.965343 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:33.965310 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:33.966570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:33.966543 2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 16:39:33.966723 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:33.966604 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 16:39:34.965677 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:34.965638 2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 16:39:34.966106 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:34.965707 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 16:39:35.965201 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:35.965164 2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 16:39:35.965398 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:35.965225 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 16:39:36.964809 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:36.964776 2574 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 16:39:36.965198 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:36.964828 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 16:39:37.005705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:37.005663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:37.081720 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:37.081684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:37.969540 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:37.969509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:37.969917 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:37.969866 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:37.970573 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:37.970553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:39:44.167595 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:44.167556 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:49.865649 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:49.865610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:49.897143 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:49.897101 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:49.897383 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:49.897335 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" podUID="217f7f7f-556a-405f-9d78-235eaf3d7150" containerName="payload-processing" containerID="cri-o://a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4" gracePeriod=30 Jul 29 16:39:49.936904 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:49.936869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:50.037093 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.037049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:50.102263 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.100579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:50.211433 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.211409 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:50.297137 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.297106 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume\") pod \"217f7f7f-556a-405f-9d78-235eaf3d7150\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " Jul 29 16:39:50.297318 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.297211 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb6lh\" (UniqueName: \"kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh\") pod \"217f7f7f-556a-405f-9d78-235eaf3d7150\" (UID: \"217f7f7f-556a-405f-9d78-235eaf3d7150\") " Jul 29 16:39:50.297489 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.297465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "217f7f7f-556a-405f-9d78-235eaf3d7150" (UID: "217f7f7f-556a-405f-9d78-235eaf3d7150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:39:50.299435 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.299405 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh" (OuterVolumeSpecName: "kube-api-access-lb6lh") pod "217f7f7f-556a-405f-9d78-235eaf3d7150" (UID: "217f7f7f-556a-405f-9d78-235eaf3d7150"). InnerVolumeSpecName "kube-api-access-lb6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:39:50.357905 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.357875 2574 generic.go:356] "Generic (PLEG): container finished" podID="217f7f7f-556a-405f-9d78-235eaf3d7150" containerID="a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4" exitCode=0 Jul 29 16:39:50.358081 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.357915 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" Jul 29 16:39:50.358081 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.357955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" event={"ID":"217f7f7f-556a-405f-9d78-235eaf3d7150","Type":"ContainerDied","Data":"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4"} Jul 29 16:39:50.358081 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.358014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5" event={"ID":"217f7f7f-556a-405f-9d78-235eaf3d7150","Type":"ContainerDied","Data":"8017ec8d3d64187fc6d7dc0270bda07f3407c0f9f6bf9aa650b4cb8e6dda9250"} Jul 29 16:39:50.358081 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.358040 2574 scope.go:117] "RemoveContainer" containerID="a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4" Jul 29 16:39:50.367028 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.367007 2574 scope.go:117] "RemoveContainer" containerID="a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4" Jul 29 16:39:50.367306 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:39:50.367270 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4\": container with ID starting with a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4 not found: ID does not exist" containerID="a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4" Jul 29 16:39:50.367371 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.367319 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4"} err="failed to get container status \"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4\": rpc error: code = NotFound desc = could not find container \"a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4\": container with ID starting with a9b7789e5f86f4024744ce35846c4843569a10c49b79b01e5459be0397043bd4 not found: ID does not exist" Jul 29 16:39:50.377105 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.377079 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:50.382617 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.382594 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1bad22-6b45b7dfb4-w2ql5"] Jul 29 16:39:50.398322 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.398288 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6lh\" (UniqueName: \"kubernetes.io/projected/217f7f7f-556a-405f-9d78-235eaf3d7150-kube-api-access-lb6lh\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:39:50.398322 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.398324 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217f7f7f-556a-405f-9d78-235eaf3d7150-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:39:50.434035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:50.434002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:52.334579 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:52.334547 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217f7f7f-556a-405f-9d78-235eaf3d7150" path="/var/lib/kubelet/pods/217f7f7f-556a-405f-9d78-235eaf3d7150/volumes" Jul 29 16:39:56.179161 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.179125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:56.246789 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.246750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:56.469720 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.469686 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:39:56.469921 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.469900 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" containerID="cri-o://d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae" gracePeriod=30 Jul 29 16:39:56.471203 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.471177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:39:56.701137 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:39:56.701101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:01.740939 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.740854 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:40:01.812996 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.812925 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8zcg\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813208 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813208 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813158 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813208 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813293 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813374 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813376 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data\") pod \"6779c7c9-a6fe-456e-bfed-157d35c57c41\" (UID: \"6779c7c9-a6fe-456e-bfed-157d35c57c41\") " Jul 29 16:40:01.813570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813523 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:40:01.813670 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813590 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:40:01.813723 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813670 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:40:01.813823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813804 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6779c7c9-a6fe-456e-bfed-157d35c57c41-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.813878 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813829 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.813878 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813829 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:40:01.813878 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.813842 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.814427 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.814396 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data" (OuterVolumeSpecName: "istio-data") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:40:01.815492 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.815468 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:40:01.815950 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.815926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg" (OuterVolumeSpecName: "kube-api-access-q8zcg") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "kube-api-access-q8zcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:40:01.815950 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.815936 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token" (OuterVolumeSpecName: "istio-token") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:40:01.816125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.816084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6779c7c9-a6fe-456e-bfed-157d35c57c41" (UID: "6779c7c9-a6fe-456e-bfed-157d35c57c41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:40:01.915171 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915137 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.915171 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915167 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zcg\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-kube-api-access-q8zcg\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.915171 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915178 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.915418 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915186 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.915418 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915194 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6779c7c9-a6fe-456e-bfed-157d35c57c41-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:01.915418 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:01.915201 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6779c7c9-a6fe-456e-bfed-157d35c57c41-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:40:02.431499 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.431465 2574 generic.go:356] "Generic (PLEG): container finished" podID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerID="d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae" exitCode=0 Jul 29 16:40:02.431675 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.431513 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" Jul 29 16:40:02.431675 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.431552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerDied","Data":"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae"} Jul 29 16:40:02.431675 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.431610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7" event={"ID":"6779c7c9-a6fe-456e-bfed-157d35c57c41","Type":"ContainerDied","Data":"23fedbfc3e8f401f74f520f5232fced6d6815615cc38c2934aedc1768698a3de"} Jul 29 16:40:02.431675 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.431640 2574 scope.go:117] "RemoveContainer" containerID="d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae" Jul 29 16:40:02.441122 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.441104 2574 scope.go:117] "RemoveContainer" containerID="92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81" Jul 29 16:40:02.450184 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.450159 2574 scope.go:117] "RemoveContainer" containerID="d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae" Jul 29 16:40:02.450494 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:40:02.450472 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae\": container with ID starting with d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae not found: ID does not exist" containerID="d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae" Jul 29 16:40:02.450596 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.450505 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae"} err="failed to get container status \"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae\": rpc error: code = NotFound desc = could not find container \"d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae\": container with ID starting with d25c7b96faa7d2131691eac7bd31b8daeae2e410903f52a7b5d9ff5a7aeee5ae not found: ID does not exist" Jul 29 16:40:02.450596 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.450534 2574 scope.go:117] "RemoveContainer" containerID="92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81" Jul 29 16:40:02.450856 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:40:02.450837 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81\": container with ID starting with 92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81 not found: ID does not exist" containerID="92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81" Jul 29 16:40:02.450929 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.450861 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81"} err="failed to get container status \"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81\": rpc error: code = NotFound desc = could not find container \"92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81\": container with ID starting with 92043fc7be3203afe0c837d1da5d41de3c9a3c17db245f2852cd6836231cec81 not found: ID does not exist" Jul 29 16:40:02.452301 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.452275 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:40:02.456359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:02.456334 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1bad22-openshift-default-657599b568-lvnz7"] Jul 29 16:40:04.340551 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:04.340518 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" path="/var/lib/kubelet/pods/6779c7c9-a6fe-456e-bfed-157d35c57c41/volumes" Jul 29 16:40:15.723590 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:15.723554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:15.797716 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:15.797653 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:15.897452 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:15.897416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:15.968999 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:15.968941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:16.463970 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:16.463932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:18.168352 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:18.168306 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:18.269056 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:18.269015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:21.365732 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:21.365698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:38.655693 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:38.655602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:58.543968 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:58.543935 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:40:58.959417 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:40:58.959374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:04.278219 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.278187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:04.349297 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.349264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:04.530346 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.530248 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:41:04.530568 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.530523 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" containerID="cri-o://c4b06038805fcace770120989e585f9a7e38e5fed3925183f8fa71671f6f9775" gracePeriod=30 Jul 29 16:41:04.563168 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.563130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:04.798055 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:04.797953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:09.803253 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.803221 2574 generic.go:356] "Generic (PLEG): container finished" podID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerID="c4b06038805fcace770120989e585f9a7e38e5fed3925183f8fa71671f6f9775" exitCode=0 Jul 29 16:41:09.803678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.803254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerDied","Data":"c4b06038805fcace770120989e585f9a7e38e5fed3925183f8fa71671f6f9775"} Jul 29 16:41:09.803678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.803316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" event={"ID":"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4","Type":"ContainerDied","Data":"febda0df61d01789a7164263b40a74ffdbc56cf0dcc447b42f02c9a9f9680e8d"} Jul 29 16:41:09.803678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.803337 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="febda0df61d01789a7164263b40a74ffdbc56cf0dcc447b42f02c9a9f9680e8d" Jul 29 16:41:09.803678 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.803352 2574 scope.go:117] "RemoveContainer" containerID="7ab2f29cfb1e667bd7f262d4e40a218bb108e0876d33b85b91669271f525cddc" Jul 29 16:41:09.806894 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.806873 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:41:09.885880 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.885849 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.885880 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.885885 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.885908 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.885945 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.885964 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8x5n\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886015 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886040 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886160 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886067 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886314 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:09.886450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886396 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:09.886450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket\") pod \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\" (UID: \"5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4\") " Jul 29 16:41:09.886607 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886572 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:09.886782 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886751 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:09.886925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886836 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.886925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886855 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.886925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886868 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.886925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886875 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:09.886925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.886881 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.888747 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.888723 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:09.888835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.888743 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n" (OuterVolumeSpecName: "kube-api-access-p8x5n") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "kube-api-access-p8x5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:09.888835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.888764 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:09.888835 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.888771 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" (UID: "5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:41:09.987881 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.987842 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.987881 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.987873 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.987881 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.987886 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8x5n\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-kube-api-access-p8x5n\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.988168 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.987898 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:09.988168 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:09.987910 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:10.810955 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:10.810925 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl" Jul 29 16:41:10.848017 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:10.847966 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:41:10.853027 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:10.853000 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-777250-openshift-default-84666fdcd6-gzgfl"] Jul 29 16:41:12.334092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:12.334061 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" path="/var/lib/kubelet/pods/5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4/volumes" Jul 29 16:41:15.473415 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:15.473382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:15.537779 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:15.537716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:15.600937 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:15.600888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:15.973368 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:15.973340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:22.246523 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:22.246486 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:41:22.247007 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:22.246747 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" containerID="cri-o://74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2" gracePeriod=30 Jul 29 16:41:22.386407 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:22.386372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:22.453358 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:22.453321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:22.678585 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:22.678500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:27.518240 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.518177 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:41:27.671422 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671383 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671433 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2k8v\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671523 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671550 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671608 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.671909 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671832 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:27.671909 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671848 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:27.671909 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.671879 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:27.672514 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert\") pod \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\" (UID: \"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9\") " Jul 29 16:41:27.672514 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672451 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:27.672514 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672495 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.672630 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672516 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.672630 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672531 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.672894 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.672870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:27.673820 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.673796 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:27.673940 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.673917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v" (OuterVolumeSpecName: "kube-api-access-b2k8v") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "kube-api-access-b2k8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:27.674113 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.674092 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:27.674173 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.674089 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" (UID: "1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773161 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773194 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2k8v\" (UniqueName: \"kubernetes.io/projected/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-kube-api-access-b2k8v\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773205 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773213 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773223 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.773260 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.773230 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:27.902914 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.902881 2574 generic.go:356] "Generic (PLEG): container finished" podID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerID="74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2" exitCode=0 Jul 29 16:41:27.903096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.902924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerDied","Data":"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2"} Jul 29 16:41:27.903096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.902932 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" Jul 29 16:41:27.903096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.903021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh" event={"ID":"1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9","Type":"ContainerDied","Data":"f5f669ce473b0cd5da0af425888d3bcd1faa4b7495a8bfdf0aefd6b2c8dd36d2"} Jul 29 16:41:27.903096 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.903048 2574 scope.go:117] "RemoveContainer" containerID="74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2" Jul 29 16:41:27.913062 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.913051 2574 scope.go:117] "RemoveContainer" containerID="95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a" Jul 29 16:41:27.921349 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.921330 2574 scope.go:117] "RemoveContainer" containerID="74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2" Jul 29 16:41:27.921599 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:41:27.921580 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2\": container with ID starting with 74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2 not found: ID does not exist" containerID="74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2" Jul 29 16:41:27.921642 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.921608 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2"} err="failed to get container status \"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2\": rpc error: code = NotFound desc = could not find container \"74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2\": container with ID starting with 74977df280afd380682c9273b2444d707a3a6d86149b95712576dc4e0e489fd2 not found: ID does not exist" Jul 29 16:41:27.921642 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.921624 2574 scope.go:117] "RemoveContainer" containerID="95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a" Jul 29 16:41:27.921846 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:41:27.921827 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a\": container with ID starting with 95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a not found: ID does not exist" containerID="95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a" Jul 29 16:41:27.921893 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.921854 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a"} err="failed to get container status \"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a\": rpc error: code = NotFound desc = could not find container \"95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a\": container with ID starting with 95c323d6bfc90ec014435ddc882549d2d745a05fed730b159d3511c568d1cb8a not found: ID does not exist" Jul 29 16:41:27.925246 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.925227 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:41:27.929176 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:27.929156 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3c2006-openshift-default-7465dcbf5b-g8tbh"] Jul 29 16:41:28.333736 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:28.333702 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" path="/var/lib/kubelet/pods/1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9/volumes" Jul 29 16:41:33.118200 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:33.118167 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv"] Jul 29 16:41:33.120841 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:33.120814 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-81ab8b-29755710-vw9rv"] Jul 29 16:41:33.719018 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:33.718966 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:33.818359 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:33.818327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:34.334172 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:34.334140 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2869b6f4-f1fe-4a8d-952a-9b58b5c64629" path="/var/lib/kubelet/pods/2869b6f4-f1fe-4a8d-952a-9b58b5c64629/volumes" Jul 29 16:41:34.770419 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:34.770380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:39.885857 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:39.885822 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:41:39.886325 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:39.886091 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" containerID="cri-o://17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8" gracePeriod=30 Jul 29 16:41:39.890057 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:39.890030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:39.949682 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:39.949644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:40.156424 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:40.156329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:44.587574 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:44.587543 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:41:44.606277 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:44.606252 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r4hgq_98be5804-426c-4fa2-883c-0e9e0208cbd4/ovn-acl-logging/0.log" Jul 29 16:41:44.780591 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:44.780557 2574 scope.go:117] "RemoveContainer" containerID="6608072eaa79500b9de60aea4d5651384425a0abc8f5377abcb48d5bf160d136" Jul 29 16:41:45.153119 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.153047 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:41:45.240618 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240627 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6g45\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240700 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240791 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.240814 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.241222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.241222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data\") pod \"04aef03e-3024-4281-a649-f425d6aebf15\" (UID: \"04aef03e-3024-4281-a649-f425d6aebf15\") " Jul 29 16:41:45.241222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.240914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:45.241222 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.241157 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:45.241411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.241224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:41:45.241411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.241298 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.241411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.241356 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.241411 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.241367 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:45.242024 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.242000 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data" (OuterVolumeSpecName: "istio-data") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:45.243044 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.243022 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45" (OuterVolumeSpecName: "kube-api-access-v6g45") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "kube-api-access-v6g45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:45.243262 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.243241 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:41:45.243462 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.243440 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token" (OuterVolumeSpecName: "istio-token") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:41:45.243511 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.243441 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04aef03e-3024-4281-a649-f425d6aebf15" (UID: "04aef03e-3024-4281-a649-f425d6aebf15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:41:45.342488 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342452 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aef03e-3024-4281-a649-f425d6aebf15-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342488 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342481 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aef03e-3024-4281-a649-f425d6aebf15-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342557 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342566 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342574 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342582 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6g45\" (UniqueName: \"kubernetes.io/projected/04aef03e-3024-4281-a649-f425d6aebf15-kube-api-access-v6g45\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:45.342705 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:45.342590 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aef03e-3024-4281-a649-f425d6aebf15-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:41:46.004965 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.004929 2574 generic.go:356] "Generic (PLEG): container finished" podID="04aef03e-3024-4281-a649-f425d6aebf15" containerID="17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8" exitCode=0 Jul 29 16:41:46.005478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.004973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" Jul 29 16:41:46.005478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.005020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerDied","Data":"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8"} Jul 29 16:41:46.005478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.005090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl" event={"ID":"04aef03e-3024-4281-a649-f425d6aebf15","Type":"ContainerDied","Data":"a4fbdd7d49b8b1241798af8e6fb54e9e8b2ff257f09a19baf2b37dea75614c56"} Jul 29 16:41:46.005478 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.005129 2574 scope.go:117] "RemoveContainer" containerID="17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8" Jul 29 16:41:46.014712 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.014697 2574 scope.go:117] "RemoveContainer" containerID="d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e" Jul 29 16:41:46.023035 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.023017 2574 scope.go:117] "RemoveContainer" containerID="17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8" Jul 29 16:41:46.023297 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:41:46.023281 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8\": container with ID starting with 17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8 not found: ID does not exist" containerID="17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8" Jul 29 16:41:46.023355 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.023307 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8"} err="failed to get container status \"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8\": rpc error: code = NotFound desc = could not find container \"17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8\": container with ID starting with 17f0decf5343cdeb767a543e6f1481c5036b7b7f84d6faad2a19816b2e6505a8 not found: ID does not exist" Jul 29 16:41:46.023355 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.023322 2574 scope.go:117] "RemoveContainer" containerID="d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e" Jul 29 16:41:46.023540 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:41:46.023521 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e\": container with ID starting with d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e not found: ID does not exist" containerID="d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e" Jul 29 16:41:46.023580 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.023547 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e"} err="failed to get container status \"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e\": rpc error: code = NotFound desc = could not find container \"d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e\": container with ID starting with d48105c2b89f49df8720292f8303c1ebe9e2dfd3276d8074ef5f59f434f92c2e not found: ID does not exist" Jul 29 16:41:46.027131 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.027106 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:41:46.030911 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.030889 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-81ab8b-openshift-default-7cfd458848-khkkl"] Jul 29 16:41:46.333966 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:46.333880 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04aef03e-3024-4281-a649-f425d6aebf15" path="/var/lib/kubelet/pods/04aef03e-3024-4281-a649-f425d6aebf15/volumes" Jul 29 16:41:50.724921 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.724885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:50.729439 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.729391 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch"] Jul 29 16:41:50.731420 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.731397 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-eccd34-29755710-d4dch"] Jul 29 16:41:50.804092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.804055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:50.862685 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.862645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:50.928452 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:50.928412 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:51.281229 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:51.281193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:52.333013 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:52.332935 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7997ee18-f2fb-42b5-8c0b-0be9651408f1" path="/var/lib/kubelet/pods/7997ee18-f2fb-42b5-8c0b-0be9651408f1/volumes" Jul 29 16:41:57.372356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.372324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:57.432218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.432181 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:57.497794 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.497755 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:41:57.498092 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.498069 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" containerID="cri-o://91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5" gracePeriod=30 Jul 29 16:41:57.522063 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.522026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:57.740296 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:57.740244 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:41:58.179190 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:41:58.179092 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b595dc6c7-qjhm5_a228d4a1-b1ae-450f-844c-d3e9b6675df6/authorino/0.log" Jul 29 16:42:02.754896 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.754867 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:42:02.807515 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807425 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807515 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cz7f\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807515 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807498 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807621 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807728 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.807799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807793 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.808047 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807814 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:42:02.808047 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807830 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data\") pod \"f5a361ef-f3ee-482d-a9ca-e42da3359011\" (UID: \"f5a361ef-f3ee-482d-a9ca-e42da3359011\") " Jul 29 16:42:02.808047 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.807897 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:42:02.809591 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.808464 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:42:02.809591 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.809388 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:42:02.810154 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810071 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 16:42:02.810154 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810135 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.810280 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810170 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.810280 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810185 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.810280 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810201 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.812026 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810830 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 16:42:02.812026 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.810904 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:42:02.813826 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.813494 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f" (OuterVolumeSpecName: "kube-api-access-7cz7f") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "kube-api-access-7cz7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 16:42:02.816519 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.816479 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5a361ef-f3ee-482d-a9ca-e42da3359011" (UID: "f5a361ef-f3ee-482d-a9ca-e42da3359011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 16:42:02.911704 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.911666 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.911704 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.911706 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a361ef-f3ee-482d-a9ca-e42da3359011-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.911913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.911721 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.911913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.911735 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cz7f\" (UniqueName: \"kubernetes.io/projected/f5a361ef-f3ee-482d-a9ca-e42da3359011-kube-api-access-7cz7f\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:02.911913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:02.911750 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a361ef-f3ee-482d-a9ca-e42da3359011-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\"" Jul 29 16:42:03.098913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.098827 2574 generic.go:356] "Generic (PLEG): container finished" podID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerID="91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5" exitCode=0 Jul 29 16:42:03.098913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.098886 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" Jul 29 16:42:03.098913 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.098900 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerDied","Data":"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5"} Jul 29 16:42:03.099179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.098990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2" event={"ID":"f5a361ef-f3ee-482d-a9ca-e42da3359011","Type":"ContainerDied","Data":"df171a346c33ee307a358916a6bc8b3d6f71f42650044fcd2d7933784ff4883e"} Jul 29 16:42:03.099179 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.099017 2574 scope.go:117] "RemoveContainer" containerID="91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5" Jul 29 16:42:03.108584 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.108565 2574 scope.go:117] "RemoveContainer" containerID="78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b" Jul 29 16:42:03.117078 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.117059 2574 scope.go:117] "RemoveContainer" containerID="91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5" Jul 29 16:42:03.117361 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:42:03.117340 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5\": container with ID starting with 91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5 not found: ID does not exist" containerID="91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5" Jul 29 16:42:03.117412 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.117370 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5"} err="failed to get container status \"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5\": rpc error: code = NotFound desc = could not find container \"91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5\": container with ID starting with 91454ada63973fa7ef75d478ae8aaa152ea532a7a5d3c00bf8750b8a63c533a5 not found: ID does not exist" Jul 29 16:42:03.117412 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.117388 2574 scope.go:117] "RemoveContainer" containerID="78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b" Jul 29 16:42:03.117630 ip-10-0-130-153 kubenswrapper[2574]: E0729 16:42:03.117613 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b\": container with ID starting with 78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b not found: ID does not exist" containerID="78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b" Jul 29 16:42:03.117667 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.117637 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b"} err="failed to get container status \"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b\": rpc error: code = NotFound desc = could not find container \"78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b\": container with ID starting with 78c7df366ced3949bcd38a28cdbc663a32a66e1d78c1f641602b52b4bc01122b not found: ID does not exist" Jul 29 16:42:03.121632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.121604 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:42:03.125321 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.125299 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-eccd34-openshift-default-777dd875fb-59nv2"] Jul 29 16:42:03.599570 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.599539 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-x7jdn_6e74ea06-8bbb-42d5-abd4-94d21eae1b44/manager/0.log" Jul 29 16:42:03.725813 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.725781 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8mmvb_c84d81c9-379b-4dc5-899e-6b2d027625bf/manager/0.log" Jul 29 16:42:03.855922 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.855837 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7679986d7-9r2ff_ac215dc4-b493-4d54-ab75-28cc0a803fa7/manager/0.log" Jul 29 16:42:03.961471 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:03.961426 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p4fbj_565bce0f-3411-4174-b34f-421d8f434a61/server/0.log" Jul 29 16:42:04.080613 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:04.080578 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7cwgg_1d38911e-b47a-4819-97b5-7c8d37cc113c/manager/0.log" Jul 29 16:42:04.186811 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:04.186731 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-7dsrn_ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0/copy-manifests/0.log" Jul 29 16:42:04.205469 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:04.205439 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-7dsrn_ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0/manager/0.log" Jul 29 16:42:04.334469 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:04.334441 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" path="/var/lib/kubelet/pods/f5a361ef-f3ee-482d-a9ca-e42da3359011/volumes" Jul 29 16:42:04.954780 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:04.954743 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755680-4886q_2cc4924d-2a06-4882-9f8e-be88f315bd1c/cleanup/0.log" Jul 29 16:42:05.065805 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:05.065776 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755695-72jj2_2dbafa0b-195a-4f7c-a7db-bb592208a5a3/cleanup/0.log" Jul 29 16:42:05.181460 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:05.181425 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755710-blztd_4da9fa63-68a9-4705-bdaf-8b187d0fdbf6/cleanup/0.log" Jul 29 16:42:06.543925 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:06.543833 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b595dc6c7-qjhm5_a228d4a1-b1ae-450f-844c-d3e9b6675df6/authorino/0.log" Jul 29 16:42:06.654927 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:06.654891 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xdc7x_1ba5342d-c3ab-43de-9d78-31369ebc2cf5/manager/0.log" Jul 29 16:42:06.768737 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:06.768709 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bmzz8_f6a07185-7099-4084-acbb-6dea09eb962a/manager/0.log" Jul 29 16:42:07.010312 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:07.010277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-768tz_6f6e115f-647c-49da-b9f2-6741b2ba8ed4/registry-server/0.log" Jul 29 16:42:07.134356 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:07.134325 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-knrzs_03bdc93b-ab6b-4744-b05e-36368a1f0bfb/manager/0.log" Jul 29 16:42:07.371999 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:07.371910 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-w797v_ffde698a-1f71-4869-aa99-3a04c92ed33b/manager/0.log" Jul 29 16:42:07.717170 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:07.717136 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h_38254bd7-29dd-40a3-8f7d-408516ea47cf/istio-proxy/0.log" Jul 29 16:42:08.057201 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:08.057118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbb74d46-6q8p2_84385d44-1b71-4ada-b28c-a24c8dc9a9bb/kube-auth-proxy/0.log" Jul 29 16:42:08.179036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:08.179003 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/1.log" Jul 29 16:42:08.865277 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:08.865246 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c4zjl_5d361af7-dcbb-41f4-9743-1da53a85359b/main/0.log" Jul 29 16:42:08.979535 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:08.979503 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-w5vkp_f0e95358-aa84-4c04-85b7-5238e09f51c0/main/0.log" Jul 29 16:42:14.908506 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:14.908467 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:42:14.992580 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:14.992547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" limit=1024 Jul 29 16:42:17.894971 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:17.894923 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w9k2g_901c05b2-86a2-4a19-9e37-99b5f4b79c26/global-pull-secret-syncer/0.log" Jul 29 16:42:17.949285 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:17.949250 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kzhbk_63d1846d-8f08-4e0e-bd53-4670efb97c85/konnectivity-agent/0.log" Jul 29 16:42:18.060829 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:18.060797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-153.ec2.internal_c7c1d29852edb376853ce02192f5c250/haproxy/0.log" Jul 29 16:42:23.525373 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:23.525337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b595dc6c7-qjhm5_a228d4a1-b1ae-450f-844c-d3e9b6675df6/authorino/0.log" Jul 29 16:42:23.552262 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:23.552232 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xdc7x_1ba5342d-c3ab-43de-9d78-31369ebc2cf5/manager/0.log" Jul 29 16:42:23.578989 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:23.578926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bmzz8_f6a07185-7099-4084-acbb-6dea09eb962a/manager/0.log" Jul 29 16:42:23.677788 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:23.677751 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-768tz_6f6e115f-647c-49da-b9f2-6741b2ba8ed4/registry-server/0.log" Jul 29 16:42:23.991468 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:23.991425 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-knrzs_03bdc93b-ab6b-4744-b05e-36368a1f0bfb/manager/0.log" Jul 29 16:42:24.120450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:24.120403 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-w797v_ffde698a-1f71-4869-aa99-3a04c92ed33b/manager/0.log" Jul 29 16:42:26.087294 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.087261 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-lzwd6_e94fa930-5921-4a69-bab2-273ff8122b6b/monitoring-plugin/0.log" Jul 29 16:42:26.115753 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.115717 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfpwj_f497917b-1860-4d56-ac85-debb1e9b6e07/node-exporter/0.log" Jul 29 16:42:26.134240 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.134218 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfpwj_f497917b-1860-4d56-ac85-debb1e9b6e07/kube-rbac-proxy/0.log" Jul 29 16:42:26.164212 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.164185 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfpwj_f497917b-1860-4d56-ac85-debb1e9b6e07/init-textfile/0.log" Jul 29 16:42:26.386450 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.386374 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/prometheus/0.log" Jul 29 16:42:26.411201 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.411177 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/config-reloader/0.log" Jul 29 16:42:26.429498 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.429476 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/thanos-sidecar/0.log" Jul 29 16:42:26.447408 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.447384 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/kube-rbac-proxy-web/0.log" Jul 29 16:42:26.474610 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.474590 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/kube-rbac-proxy/0.log" Jul 29 16:42:26.493061 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.493039 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/kube-rbac-proxy-thanos/0.log" Jul 29 16:42:26.511730 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.511713 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec0e087f-1f24-472c-8373-1fabcd53b937/init-config-reloader/0.log" Jul 29 16:42:26.537426 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.537386 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-c245l_06699a36-83be-48d8-b653-9d549d117cb5/prometheus-operator/0.log" Jul 29 16:42:26.551632 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:26.551601 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-c245l_06699a36-83be-48d8-b653-9d549d117cb5/kube-rbac-proxy/0.log" Jul 29 16:42:27.903283 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903250 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46"] Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903673 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903687 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903699 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903705 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903713 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903718 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903726 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.903728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903731 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903738 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903743 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903752 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903756 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903764 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903769 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903777 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903782 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903792 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903799 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903806 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217f7f7f-556a-405f-9d78-235eaf3d7150" containerName="payload-processing" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903811 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="217f7f7f-556a-405f-9d78-235eaf3d7150" containerName="payload-processing" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903867 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903874 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903881 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903887 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a361ef-f3ee-482d-a9ca-e42da3359011" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903891 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="217f7f7f-556a-405f-9d78-235eaf3d7150" containerName="payload-processing" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903899 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcf70e7-6e2c-43c6-9a9b-8ce1aa28ecb9" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903905 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903909 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6779c7c9-a6fe-456e-bfed-157d35c57c41" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903916 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04aef03e-3024-4281-a649-f425d6aebf15" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903921 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" Jul 29 16:42:27.904032 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.903926 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a10efdf-b2e2-4f6c-88b8-8a1a8513fcf4" containerName="istio-proxy" Jul 29 16:42:27.906942 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.906923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:27.909722 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.909695 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqvb7\"/\"openshift-service-ca.crt\"" Jul 29 16:42:27.909831 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.909740 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kqvb7\"/\"default-dockercfg-hb6xm\"" Jul 29 16:42:27.910818 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.910804 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqvb7\"/\"kube-root-ca.crt\"" Jul 29 16:42:27.915134 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.915110 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46"] Jul 29 16:42:27.937234 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.937206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-proc\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:27.937234 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.937231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-lib-modules\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:27.937394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.937288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ml7\" (UniqueName: \"kubernetes.io/projected/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-kube-api-access-b8ml7\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:27.937394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.937328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-podres\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:27.937394 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:27.937376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-sys\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.038640 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ml7\" (UniqueName: \"kubernetes.io/projected/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-kube-api-access-b8ml7\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.038823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-podres\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.038823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-sys\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.038823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-proc\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.038823 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-lib-modules\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.039036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-sys\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.039036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-podres\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.039036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-proc\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.039036 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.038881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-lib-modules\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.051728 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.051706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ml7\" (UniqueName: \"kubernetes.io/projected/4f7a3371-3a99-4bdf-bb32-c13acea2bacf-kube-api-access-b8ml7\") pod \"perf-node-gather-daemonset-vqz46\" (UID: \"4f7a3371-3a99-4bdf-bb32-c13acea2bacf\") " pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.217681 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.217651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:28.554527 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:28.554500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46"] Jul 29 16:42:28.556455 ip-10-0-130-153 kubenswrapper[2574]: W0729 16:42:28.556421 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f7a3371_3a99_4bdf_bb32_c13acea2bacf.slice/crio-ddc33a499ef2d7051453c0fabbd5ca838e74bcc0f77b01979c9c72eb613fc885 WatchSource:0}: Error finding container ddc33a499ef2d7051453c0fabbd5ca838e74bcc0f77b01979c9c72eb613fc885: Status 404 returned error can't find the container with id ddc33a499ef2d7051453c0fabbd5ca838e74bcc0f77b01979c9c72eb613fc885 Jul 29 16:42:29.250799 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:29.250763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" event={"ID":"4f7a3371-3a99-4bdf-bb32-c13acea2bacf","Type":"ContainerStarted","Data":"e15e22da4954d88970d3b815160bfe4e7240f69f729f3f399c08cfeefe5426f0"} Jul 29 16:42:29.251277 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:29.250900 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:29.251277 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:29.250956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" event={"ID":"4f7a3371-3a99-4bdf-bb32-c13acea2bacf","Type":"ContainerStarted","Data":"ddc33a499ef2d7051453c0fabbd5ca838e74bcc0f77b01979c9c72eb613fc885"} Jul 29 16:42:29.266516 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:29.266471 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" podStartSLOduration=2.266456479 podStartE2EDuration="2.266456479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 16:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 16:42:29.265573186 +0000 UTC m=+3345.504130397" watchObservedRunningTime="2026-07-29 16:42:29.266456479 +0000 UTC m=+3345.505013688" Jul 29 16:42:30.655992 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:30.655948 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rwjjb_6cb7b2da-358a-4963-8d3c-0d31f5db64bb/dns/0.log" Jul 29 16:42:30.673486 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:30.673460 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rwjjb_6cb7b2da-358a-4963-8d3c-0d31f5db64bb/kube-rbac-proxy/0.log" Jul 29 16:42:30.736946 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:30.736911 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-994kz_af4961fd-5883-4b04-8eb0-1c950898d3ab/dns-node-resolver/0.log" Jul 29 16:42:31.297218 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:31.297169 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-849bb45f95-6djs8_30bd5209-f3e4-4e63-8378-f1330a3a821d/registry/0.log" Jul 29 16:42:31.341049 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:31.341023 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ptlz_2fb8fbe2-7a23-421c-ab82-7de86ba3d6b4/node-ca/0.log" Jul 29 16:42:32.289131 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:32.289100 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h_38254bd7-29dd-40a3-8f7d-408516ea47cf/istio-proxy/0.log" Jul 29 16:42:32.484738 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:32.484690 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbb74d46-6q8p2_84385d44-1b71-4ada-b28c-a24c8dc9a9bb/kube-auth-proxy/0.log" Jul 29 16:42:32.515205 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:32.515174 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/0.log" Jul 29 16:42:32.568296 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:32.568209 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/1.log" Jul 29 16:42:33.461890 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:33.461858 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vwgt7_750ea16e-e156-4180-b5f8-067086f46f82/serve-healthcheck-canary/0.log" Jul 29 16:42:34.069973 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:34.069939 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4h77_0d5aebed-f552-4edf-ac20-82310dd6cf44/kube-rbac-proxy/0.log" Jul 29 16:42:34.088691 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:34.088653 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4h77_0d5aebed-f552-4edf-ac20-82310dd6cf44/exporter/0.log" Jul 29 16:42:34.107839 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:34.107811 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4h77_0d5aebed-f552-4edf-ac20-82310dd6cf44/extractor/0.log" Jul 29 16:42:35.265694 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:35.265667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kqvb7/perf-node-gather-daemonset-vqz46" Jul 29 16:42:36.211277 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:36.211248 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-x7jdn_6e74ea06-8bbb-42d5-abd4-94d21eae1b44/manager/0.log" Jul 29 16:42:36.580683 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:36.580593 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8mmvb_c84d81c9-379b-4dc5-899e-6b2d027625bf/manager/0.log" Jul 29 16:42:36.829697 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:36.829666 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7679986d7-9r2ff_ac215dc4-b493-4d54-ab75-28cc0a803fa7/manager/0.log" Jul 29 16:42:36.849535 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:36.849455 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p4fbj_565bce0f-3411-4174-b34f-421d8f434a61/server/0.log" Jul 29 16:42:36.963172 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:36.963148 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7cwgg_1d38911e-b47a-4819-97b5-7c8d37cc113c/manager/0.log" Jul 29 16:42:37.099125 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:37.099096 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-7dsrn_ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0/manager/0.log" Jul 29 16:42:37.114406 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:37.114347 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75b794959f-7dsrn_ea3e5bee-1e34-46da-b5e6-23a9c3fb34d0/copy-manifests/0.log" Jul 29 16:42:37.997687 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:37.997646 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h_38254bd7-29dd-40a3-8f7d-408516ea47cf/istio-proxy/0.log" Jul 29 16:42:38.170162 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:38.170087 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbb74d46-6q8p2_84385d44-1b71-4ada-b28c-a24c8dc9a9bb/kube-auth-proxy/0.log" Jul 29 16:42:38.187781 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:38.187758 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/0.log" Jul 29 16:42:38.241482 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:38.241451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/1.log" Jul 29 16:42:39.689208 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:39.689126 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-w225h_38254bd7-29dd-40a3-8f7d-408516ea47cf/istio-proxy/0.log" Jul 29 16:42:39.856634 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:39.856604 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bbb74d46-6q8p2_84385d44-1b71-4ada-b28c-a24c8dc9a9bb/kube-auth-proxy/0.log" Jul 29 16:42:39.874677 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:39.874645 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/0.log" Jul 29 16:42:39.929701 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:39.929668 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-68snr_ecdbcfec-1562-4f11-ad92-b72b5b6681e4/istio-proxy/1.log" Jul 29 16:42:43.377331 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:43.377300 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-72mct_9a4196b4-9930-446b-88b1-ebadbd992e83/migrator/0.log" Jul 29 16:42:43.398116 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:43.398089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-72mct_9a4196b4-9930-446b-88b1-ebadbd992e83/graceful-termination/0.log" Jul 29 16:42:44.808287 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:44.808252 2574 scope.go:117] "RemoveContainer" containerID="3b97a0532eb649eed2021ebe7ceb6bd35f801b84c3d99991b6512e8934d5ca35" Jul 29 16:42:44.998625 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:44.998595 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5cv4_648d11ed-cc61-426a-913f-cde108d6b63c/kube-multus-additional-cni-plugins/0.log" Jul 29 16:42:45.018664 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:45.018631 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5cv4_648d11ed-cc61-426a-913f-cde108d6b63c/egress-router-binary-copy/0.log" Jul 29 16:42:45.036578 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:45.036544 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5cv4_648d11ed-cc61-426a-913f-cde108d6b63c/cni-plugins/0.log" Jul 29 16:42:45.054596 ip-10-0-130-153 kubenswrapper[2574]: I0729 16:42:45.054559 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s5cv4_648d11ed-cc61-426a-913f-cde108d6b63c/bond-cni-plugin/0.log"